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Summary rel 2012-No. 112 

A descriptive analysis of state- 
supported formative assessment 
initiatives in New York and Vermont 



This study examines two state-supported 
formative assessment initiatives that 
promote a consensus definition of for- 
mative assessment endorsed by the 
Council of Chief State School Officers. It 
describes the primary components of the 
two initiatives and the strategies that 
state, district, and school leaders report 
using to support implementation of each 
initiative. 

Education policymakers and practitioners are 
increasingly interested in formative assess- 
ment, in part reflecting widespread reports in 
the education press about formative assess- 
ment’s potential for enhancing student learn- 
ing. As schools and districts work to improve 
student learning outcomes, demand has grown 
for more information on state and local efforts 
to promote formative assessment and on the 
strategies that support its implementation. 

A wide range of tools and practices are labeled 
“formative assessment,” and the broad use of 
this term has created uncertainty about just 
what it entails. To resolve some of this confu- 
sion, a national group of researchers, convened 
by the Council of Chief State School Officers 
(CCSSO), arrived at a consensus definition of 
formative assessment as “a process used by 
teachers and students during instruction that 
provides feedback to adjust ongoing teaching 



and learning to improve students’ achievement 
of intended instructional outcomes” (McMa- 
nus 2008, p. 3). To provide insights into how 
this consensus definition of formative assess- 
ment has been put into practice, this study ex- 
amines two state-supported initiatives (in New 
York and Vermont) intended to promote the 
CCSSO’s definition of formative assessment. 

Two research questions guided this study: 

• In Northeast and Islands Region states 
where state education agencies are sup- 
porting formative assessment initiatives, 
what are the primary components of each 
initiative? 

• What strategies do state, district, and 
school leaders report using to support 
implementation of formative assessment 
initiatives? 

This study relies on publicly available infor- 
mation and interviews to answer these ques- 
tions. The publicly available information used 
includes websites and online documents such 
as official statements from state education 
agencies describing each initiative, progress 
reports on initiative implementation, interim 
or summary studies on each initiative, and 
documents from initiative leaders and their 
partners. This information was complemented 
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with semistructured interviews conducted 
with two state, two district, and two school 
leaders knowledgeable about each initiative. 

The New York State Formative Assessment 
Technical Assistance Study was a pilot project 
in a large urban district that began in spring 
2007 and ended in summer 2010. The Vermont 
Formative Assessment Project, launched in 
2006/07, is an ongoing initiative that started 
with 9 schools the first year and expanded to 
55 schools by 2008. 

The study describes eight primary components 
for each initiative: 

• Goals. Both initiatives aimed to improve 
student achievement through teacher use 
of formative assessment practices. Both 
initiatives also aimed to install, sustain, 
and spread formative assessment practices 
beyond pilot sites. Other implementation 
goals differed between the two initiatives. 

• Origins. The two initiatives had different 
origins. The New York initiative emerged 
from a single district’s request to the state 
education agency for support in develop- 
ing capacity to train teachers in forma- 
tive assessment. The Vermont initiative 
emerged from the state education agency’s 
desire to help schools throughout the state 
adopt formative assessment. 

• Leaders and roles. In New York, a single 
pilot district served as the primary leader 
of the initiative, while the state education 
agency supported the initiative financially 
and brought external technical assistance 
to the district. In Vermont, the state edu- 
cation agency led the initiative, drawing 



on an external vendor for training during 
the first year and on published training 
materials in subsequent years. 

• Use of external partners. Both initiatives 
relied on external partners for support, 
but roles and costs differed. In New York, 
external technical assistance providers 
helped develop the initiative and provided 
support for the full three years of the 
initiative. In Vermont, an external vendor 
provided direct support only during the 
first year. The state built its capacity to 
sustain and expand the initiative in subse- 
quent years without ongoing support from 
the external vendor. 

• Targeted participants. The New York 
initiative focused on building the capacity 
of grade 4 and 5 teachers in formative as- 
sessment for math instruction during the 
first year, and participation was expanded 
to teachers in other grades in subsequent 
years. The Vermont initiative did not have 
a specific grade or content area focus. 

• Funding. The New York initiative was 
funded through a three-year federal Math- 
ematics and Science Partnership grant 
administered through the state. When 
the grant ended, so did the initiative. The 
Vermont initiative was funded by a variety 
of federal funds during its first year and 
by school and district funds in subsequent 
years. 

• Professional development. Both initiatives 
provided initial and ongoing professional 
development to teachers in formative 
assessment. Training came from differ- 
ent sources, and each initiative provided 
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different levels of ongoing training and 
support to teacher leaders, principals, 
district officials, and state leaders. 

• Formal evaluation. Both initiatives were 
formally evaluated. New York’s evalua- 
tion efforts involved both quantitative and 
qualitative research methods, while Ver- 
mont’s involved only qualitative methods. 
Early evaluation results suggested that 
both initiatives were successful in help- 
ing teachers gain a better understanding 
of formative assessment and incorporate 
formative assessment strategies in their 
regular classroom instruction. In Ver- 
mont, these findings encouraged state 
education agency leaders to continue the 
initiative and strategize scaling efforts. 

State, district, and school leaders who were 
interviewed described a variety of strategies 
for achieving three implementation goals: get- 
ting teachers to adopt and integrate formative 
assessment practices into regular instruction, 
supporting and sustaining teacher use of for- 
mative assessment, and spreading or scaling 
use of formative assessment beyond pilot 
populations. Through analysis of the interview 
data, the study team identified nine sets of 
strategies that were present in both initiatives: 

• Providing training by well known experts 
and credible evidence of the benefits of 
formative assessment. 

• Creating a safe environment for teachers 
to try out new practices. 



• Aligning initiative reforms with existing 
contexts and concurrent training efforts. 

• Collaborating early with local leaders. 

• Employing both voluntary and mandatory 
participation approaches. 

• Providing ongoing training and support 
for teachers and others at different levels 
of the system. 

• Establishing accountability and moni- 
toring methods for sustaining initiative 
implementation. 

• Building independent state and district 
capacity to sustain and spread teacher 
training. 

• Harnessing enthusiasm at every level of 
the system to sustain and spread formative 
assessment. 

This study details two state-supported initia- 
tives that have worked to promote a consensus 
definition of formative assessment. Bindings 
from this study offer exploratory ideas for 
future research and practice. 

Reference 

McManus, S. (2008). Attributes of effective forma- 
tive assessment. Retrieved December 29, 2008, 
from http://www.ccsso.org/publications/de- 
tails.cfm?PublicationID=362. 
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This study examines 
two state-supported 
formative 
assessment 
initiatives that 
promote a 

consensus definition 
of formative 
assessment 
endorsed by the 
Council of Chief 
State School 
Officers. It describes 
the primary 
components of 
the two Initiatives 
and the strategies 
that state, district, 
and school 
leaders report 
using to support 
Implementation 
of each initiative. 



WHY THIS STUDY? 

Interest in formative assessment has been growing 
among education policymakers and practitioners 
as evidenced by frequent reports in the education 
press on formative assessment’s potential for im- 
proving student learning (Chappuis and Chappuis 
2007/2008; Tanney et al. n.d.). Since passage of 
the No Child Left Behind Act of 2001, the federal 
government has required states to design and 
administer accountability systems that set annual 
targets for student performance on large-scale as- 
sessments that schools and districts are expected 
to reach. As educators work to help schools and 
districts improve student learning outcomes, 
interest in learning about strategies such as forma- 
tive assessment has increased. 

Yet it is not always clear what education vendors 
and practitioners mean when they talk about for- 
mative assessment. The term is used to identify a 
wide range of tools and practices (Chappuis 2005; 
McMillan 2007; Tanney et al. n.d.). Convened by 
the Council of Chief State School Officers (CCSSO) 
to clarify the term, a group of national educa- 
tion leaders and researchers reached a consensus 
definition of formative assessment as “a process 
used by teachers and students during instruction 
that provides feedback to adjust ongoing teaching 
and learning to improve students’ achievement 
of intended instructional outcomes” (McManus 
2008, p. 3; see box 1). 



Importance to the region 

In line with heightened interest nationwide, the 
Governing Board of the Regional Educational 
Laboratory (REL) Northeast and Islands re- 
quested information on formative assessment 
initiatives within the region. A state education 
leader from Rhode Island also expressed an 
interest in initiatives receiving state support, 
asking for information on how state education 
agencies have built their capacity to support ini- 
tiatives such as formative assessment that involve 
classroom-level activities. State education agen- 
cies have traditionally monitored and enforced 
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BOX 1 

Abbreviations and definitions 

Accountability and Assessment 
Comprehensive Center (AACC). A 
federally funded center that imple- 
ments and evaluates assessment and 
accountability systems for states and 
districts working to meet the target 
of proficiency for all students under 
the No Child Left Behind Act of 2001. 
AACC was an external partner for the 
New York initiative, (www.aacomp- 
center.org/cs/aacc/print/htdocs/aacc/ 
about.htm) 

Educational services agencies. Part- 
nerships of school districts, unions, 
institutions of higher education, and 
service providers in Vermont estab- 
lished to offer professional develop- 
ment to meet the regional needs of 
Vermont educators. (http://education. 
vermont.gov/new/html/prodev/ 
resources.html) 

Educational Testing Service (ETS). A 
nonprofit organization that creates 
education assessments, conducts 
education research, and develops 
services and products on a range of 
issues in elementary, secondary, and 
postsecondary education. Its Keep- 
ing Learning on Track interactive 
professional development program 
is used in the Vermont initiative. 

(w ww. ets . org/ab out/who) 



Eormative assessment. As defined by 
a national group of education lead- 
ers and researchers and endorsed 
by the Council of Chief State School 
Officers: “a process used by teachers 
and students during instruction that 
provides feedback to adjust ongoing 
teaching and learning to improve 
students’ achievement of intended 
instructional outcomes” (McManus 
2008, p. 3). Formative assessment dif- 
fers from other types of assessment 
in three key ways: it is an ongoing 
process rather than a singular event 
(McManus 2008), it requires active 
and frequent interaction between 
teachers and students (Black et al. 
2004, p. 47), and it is used to adapt 
instruction to student needs (Black 
and Wiliam 1998a). 

Keeping Learning on Track (KLT). An 
interactive professional development 
program and training curriculum, 
developed by ETS and used in the 
Vermont formative assessment initia- 
tive, trains teachers in formative as- 
sessment through school-embedded 
teacher learning communities, (www. 
ets.org/Media/Campaign/12652/ 
about.html) 

Mathematics and Science Partner- 
ships. A program run by the U.S. 
Department of Education that seeks 
to increase the academic achievement 
of students in math and science by 



increasing the content knowledge and 
teaching skills of classroom teachers 
through formula-funded grants to 
states. 

New York Comprehensive Center 
(NYCC). One of sixteen comprehen- 
sive centers funded by the U.S. De- 
partment of Education to develop the 
capacity of state education agencies 
to work with districts and schools to 
improve student achievement, (www. 
nycomprehensivecenter.org/about/) 

No Child Left Behind (NCLB) Act of 
2001. An act of the U.S. Congress 
that aims to improve the academic 
achievement of all students through 
increased state and local account- 
ability, parental choice, local control 
and flexibility, and an emphasis on 
doing what works based on scientific 
research. The NCLB Act applies to 
all schools and districts that receive 
Title I funds. 

Teacher learning community (TEC). 
Professional learning communities 
used under the Vermont forma- 
tive assessment initiative. Groups of 
teachers meet monthly to learn and 
improve the skills needed to imple- 
ment formative assessment in their 
classrooms. Each community is led 
by a trained TLC leader/facilitator. 
(http://education.vermont.gov/new/ 
html/pgm_assessment.html) 



federal and state education laws, but they have 
not commonly been staffed to provide direct 
services and supports to schools and teachers 
(Lusi 1997; Sunderman and Orfield 2006). To 
shed light on these topics, this study examines 
two state education agency initiatives within the 
region that promote practices that conform with 
the CCSSO’s consensus definition of formative 



Study questions 

This study considers the following two research 
questions: 

• In Northeast and Islands Region states where 
state education agencies are supporting 
formative assessment initiatives, what are the 
primary components of each initiative? 



assessment. 
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• What strategies do state, district, and school 
leaders report using to support implementa- 
tion of formative assessment initiatives? 

Formative assessment initiatives were defined as 
formal, organized programs that promote teaching 
and learning practices consistent with the CCSSO’s 
definition. The primary components of formative 
assessment initiatives (inputs and resources that 
constitute the basic features of an initiative; Weiss 
1998) selected for examination include the goals, 
origins, leaders and roles, use of external part- 
ners, targeted participants, funding, professional 
development, and formal evaluation. For imple- 
mentation the study focused on strategies that 
state, district, and school leaders reported using to 
promote the adoption, sustainability, and spread 
of each formative assessment initiative. 

The study focused on two formative assessment 
initiatives in New York and Vermont that fit the 
study’s selection criteria: 

• Each was identified by the state as a project or 
coordinated effort to build capacity in forma- 
tive assessment within the state. 

• Each has promoted a set of practices consis- 
tent with the definition of formative assess- 
ment endorsed by the CCSSO. 

• Each has been supported by the state educa- 
tion agency. 

• Each is unique to the state and has not been 
part of a larger national initiative. 

• Each has been implemented in at least one 
district within the state for at least one year. 

Study methods are summarized in box 2 and 
detailed in appendix A. The teacher practices pro- 
moted by each initiative are in appendix B, and the 
primary components of each initiative are listed in 
appendix C. To provide context for the study and 
illustrate efforts to resolve some of the ambiguity 
concerning the concept of formative assessment. 



appendix D briefly overviews the literature on 
formative assessment and student learning, dif- 
ficulties in promoting its use among teachers, and 
strategies to promote sustained and widespread 
use. 



FINDINGS 

This study describes the primary components of 
two initiatives that have received state support to 
promote formative assessment practices — the New 
York State Formative Assessment Technical Assis- 
tance Study (NY-FATA) and the Vermont Forma- 
tive Assessment Project (VT-FAP). It also reports 
how a small sample of state, district, and school 
leaders describe their efforts to promote three 
implementation goals shared by both initiatives: 
adoption, sustainability, and spread. Nine sets of 
implementation strategies were identified from the 
analysis of interview data. Interviewees suggested 
that all of these overarching strategies (and most 
substrategies) were employed in an effort to achieve 
at least two of the three implementation goals. 



In Northeast and Islands Region states where 
state education agencies are supporting 
formative assessment initiatives, what are the 
primary components of each initiative? 



Both NY-FATA and VT-FAP have been state 
education agency-sponsored initiatives intended 
to build local capacity in formative assessment 
practices that are con- 
sistent with the CCSSO 
definition and meet the 
other study eligibility 
criteria (see appendix A). 

The following compo- 
nents were examined for 
each initiative: 



Goals. Purposes of 
the initiative, includ- 
ing program goals 
and implementation 
goals (see box 3). 



This study describes the 
primary components 
of two initiatives that 
have received state 



support to promote 
formative assessment 
practices — the New 
York State Formative 



Assessment Technical 
Assistance Study and 
the Vermont Formative 
Assessment Project 
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BOX 2 

Summary of study methods and 
limitations 

Study sample, data sources, and 
analysis. The study team collected 
state-level data and data from 
one district and one participating 
school in each state. The districts 
and schools were nominated hy 
the state education agency or other 
initiative leaders for displaying a 
strong capacity to work with the 
state to achieve initiative goals or 
for implementing the initiative 
most fully. Data were collected 
from publicly available information 
(including documents received from 
initiative leaders and their partners) 
and interviews. Publicly available 
information included websites and 
documents available online, such 
as official state education agency 
statements describing each initia- 
tive; progress reports; and interim 
or summary studies on each initia- 
tive. Interviews were semistructured 
and conducted with two state, two 
district, and two school leaders for 
each initiative (see appendix E for 
the interview protocols). 



Data matrices were used to guide, 
organize, and analyze the data from 
public sources. Interviews were 
digitally recorded, transcribed, 
and electronically coded using the 
qualitative data analysis software 
package ATLAS.ti (appendix F lists 
the interview data codes). Questions 
on missing, unclear, or contradic- 
tory data were sent to interviewees 
for clarification. Patterns in initiative 
components and implementation 
strategies were identified through 
iterative discussions among study 
team members and verification of 
interview transcripts and public 
document data. 

The study team complied with In- 
stitutional Review Board policy and 
guidelines to ensure that interview 
subjects were informed of their rights 
as study participants. The study team 
also worked to ensure that the infor- 
mation interviewees provided was 
kept secure and confidential during 
data collection, analysis, and report 
drafting. 

A full discussion of the study’s 
methodology is in appendix A. 



Study limitations. This study is 
limited by the small number of 
respondents interviewed and by the 
collection of data from only one dis- 
trict and school participating in each 
initiative. Recommended by initiative 
leaders as sites that had displayed 
a strong capacity to work with the 
state or where the initiative had been 
implemented most fully, the district 
and school selected for examination 
in each state are unlikely to repre- 
sent typical participating sites. The 
knowledge and opinions of the two 
state leaders who were interviewed 
might not represent the knowledge 
or perspectives of all state educa- 
tion agency staff involved with each 
initiative, and the information and 
opinions shared by the two district 
and school leaders interviewed 
in each state might not represent 
those of other initiative leaders and 
participants. This study presents the 
perspectives of interviewees on initia- 
tive implementation strategies as an 
exploratory step for further research. 
It does not provide evidence of the 
effectiveness of either of these initia- 
tives or of their impact on school, 
teacher, or student outcomes. 



• Origins. Description of how the initiative 
started. 

• Leaders and roles. In-state public education 
agencies that led or supported the initiative 
and their roles. 

• Use of external partners. Roles of external 
partners that contributed to the initiative. 

• Targeted participants. Types of schools, grade 
levels, and teacher content areas targeted for 
inclusion in the initiative. 



• Funding. State-administered federal grants 
and local resources supporting the initiative. 

• Professional development. Types of teacher 
professional development and training for 
school, district, and state leaders that were of- 
fered to support initiative implementation. 

• Formal evaluations. Data collected and 
reports generated to assess initiative progress 
and outcomes. 

The following sections describe each initiative by 

component (appendix C details each component). 
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BOX 3 

Program and implementation goals 

Some researchers distinguish between activities related 
to program intervention and program implementation. 
Fixsen et al. (2005, pp. 5-6) define intervention activi- 
ties as “treatment or prevention efforts at the consumer 
level” and implementation activities as “efforts to 
incorporate a program or practice at the community, 
agency, or practitioner levels.” They suggest that imple- 
mentation activities may unfold in a common set of 
stages, including program adoption, initial implemen- 
tation, full operation, innovation, and sustainability. 
Activities related to scaling up or spreading programs 
may follow this last stage. This study uses the following 
definitions: 

Program goals (such as higher student achievement) 
are related to the possible outcomes of program inter- 
vention activities, which involve interactions between 
students and teachers (such as interactions that occur 
in formative assessment). 

Implementation goals (such as teacher adoption, sus- 
tained use, or spread of formative assessment prac- 
tices) are related to the possible outcomes of program 
implementation activities, which involve interactions 
among teachers, teacher communities, teacher coaches, 
school principals, district administrators, state educa- 
tion agency officials, and external partners to further 
program goals. 

New York Formative Assessment Technical Assis- 
tance Study. NY-FATA was a three-year pilot proj- 
ect in a large urban district that began in spring 
2007 and ended in summer 2010. It was sponsored 
by the New York State Education Department. 

Goals. The NY-FATA initiative had a mix of pro- 
gram and implementation goals. Program goals 
included higher student achievement through 
teacher use of formative assessment practices. 
Implementation goals included higher quality 
math instruction through strong professional 
development in math content, pedagogy, and 
formative assessment. District and state education 



leaders hoped to build district capacity to spread 
formative assessment across the pilot district. They 
also hoped to promote a greater understanding of 
formative assessment among state education of- 
ficials in different departments to increase support 
for formative assessment throughout the state. 

Origins. Education leaders from a large urban 
school district were interested in strengthen- 
ing teachers’ math instructional skills. Because 
outside experts had advised the district to improve 
student progress monitoring, education leaders 
were also interested in building teacher capacity 
to conduct fine-grained, day-to-day monitoring of 
student learning needs and to adjust instruction 
to address those needs. State education agency 
officials offered support by providing access to two 
federal education technical assistance centers with 
expertise in formative assessment. State education 
leaders were interested in supporting the district’s 
efforts to learn whether the approach it developed 
could serve as an effective school improvement 
strategy for other districts. State officials encour- 
aged the district to pursue funding through the 
state to support capacity-building. 

Leaders and roles. District leaders had a major role 
in conceptualizing and designing the initiative. 
They also provided oversight for implementa- 
tion and identified ways to sustain and scale the 
initiative within the district. Prior to the design 
and development of the initiative, the New York 
State Education Department played an instrumen- 
tal role by connecting the district with external 
technical assistance experts and funding. Once the 
initiative was launched, state officials assumed an 
observer role, keeping abreast of implementation 
progress through regular interactions with the 
New York Comprehensive Center (NYCC). 

Use of external partners. The NYCC and the As- 
sessment and Accountability Comprehensive 
Center (AACC) helped district leaders frame and 
design the formative assessment initiative and 
provided professional development on formative 
assessment to district staff. Consultants from both 
centers also supported state education leaders by 
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The NY-FATA initiative 
aimed to promote 
teacher use of formative 
assessment by providing 
a variety of professional 
development offerings 
year-round. Different 
types of training and 
support were offered 
for teachers, math 
coaches, district leaders, 
and state education 



agency officials 



informing them about implemen- 
tation progress and holding regu- 
lar training meetings to build state 
officials’ knowledge and under- 
standing of formative assessment. 

Targeted participants. The initia- 
tive targeted elementary schools 
within the district and specific 
grade levels and content areas 
within those schools. 



• Targeted schools. In 2007, 
district officials met with elemen- 
tary school principals to inform 
them of the initiative; principals of 10 of the 
20 schools in the district volunteered their 
schools to be part of the pilot initiative, but 
only 9 schools continued in the program due 
to teacher resistance at one of the schools. 



• Targeted grade levels. The initiative targeted 

grade 4 and 5 teachers, and participation of 
teachers at those grade levels was mandatory. 
At one school, the principal found that kin- 
dergarten and grade 1 teachers were more in- 
terested in participating than were the grade 4 
and 5 teachers, so the school was accepted 
into the initiative with the participation of the 
kindergarten and grade 1 teachers instead. 
After the first year, participation in all schools 
was expanded to teachers of other grade levels 
based on teacher interest. 



was free. The district paid the salaries of math 
coaches — district staff who are tenured teach- 
ers on special assignment and who were in place 
before NY-FATA started. 

Professional development. The initiative aimed to 
promote teacher use of formative assessment by 
providing a variety of professional development of- 
ferings year-round. Different types of training and 
support were offered for teachers, math coaches, 
district leaders, and state education agency 
officials. 

• For teachers. In the first year, math coaches — 
who received training in formative assessment 
during formal monthly workshops provided 
by the NYCC — were responsible for im- 
mediately transferring their newly acquired 
knowledge to math teachers in monthly train- 
ing sessions for grade-level teams of teachers. 
These sessions mirrored the formal training 
workshops the coaches had received. In the 
second and third years, math coaches modi- 
fied training to address the needs of specific 
schools and teachers. Individual or small- 
group coaching was used to impart skills in 
modifying instruction based on analysis of 
student work. Coaches modeled lessons using 
formative assessment practices. Several math 
coaches also created 30-hour afterschool 
professional development courses in formative 
assessment open to teachers across the district 
(see appendix C for more details). 



• Targeted content area. The initiative promoted 
the use of formative assessment for math 
instruction. 

Funding. District funding for the three-year 
initiative came from a federal Title II-B Mathemat- 
ics and Science Partnership grant administered 
through the New York State Education Depart- 
ment. NY-FATA was one element of the district’s 
grant proposal. The grant paid for materials, sub- 
stitute teachers, and other training-related costs. 
Professional development provided by the NYCC 
and AACC for district and state education leaders 



• For teacher leaders (math coaches). With 
input from district leaders and support from 
the AACC, NYCC staff provided professional 
development training to district elemen- 
tary school math coaches in the theory and 
practice of classroom-level formative assess- 
ment and on how to transfer this knowledge 
to teachers. Math coaches received training 
through different formats over three years. 
During the first two summers, math coaches 
attended a mandatory summer training 
institute on formative assessment led by a 
nationally recognized formative assessment 
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scholar from the AACC, with support from 
the NYCC. During the first school year, NYCC 
staff provided mandatory monthly train- 
ing workshops to math coaches. During the 
second year, NYCC staff, district leaders, and 
other math coaches observed coaches work- 
ing with teachers in classrooms and provided 
feedback. During the third year, NYCC staff 
provided advice and technical assistance to 
coaches by phone and email, as needed. 

• For principals. During the first two years 
of the initiative, principals of participating 
schools were invited to the summer training 
sessions.' During the school year, principals 
or their designees were invited to attend 
meetings every few months with district 
leaders and NYCC staff, to inform them of 
progress and solicit their input on initiative 
implementation. 

• For district leaders. District leaders partici- 
pated in the two summer training institutes 
and the monthly training workshops for math 
coaches during the first year. District leaders 
also met regularly with NYCC staff to plan the 
training for math coaches in the second year 
and to discuss strategies for implementing the 
initiative across the district. 

• For state education agency leaders. During the 
first two years, state education agency leaders 
were invited to attend the summer training 
sessions.^ Also during the first two years, 
NYCC worked with state initiative leaders 
from the Office of Curriculum, Instruction, 
and Instructional Technology and other New 
York State Education Department offices to 
build a shared understanding of formative 
assessment and to brainstorm ways the state 
education agency could support the spread of 
formative assessment to more districts and 
schools. 

Formal evaluations. Researchers from the 

State University of New York system evaluated 

NY-FATA using both quantitative and qualitative 



research (a mixed methodology design). Addition- 
ally, NYCC and the New York State Education 
Department produced a white paper on the first 
two years of the initiative drawing from survey 
data and the Mathematics and Science Partner- 
ship grant evaluation (Tanney et al. n.d.). The 
white paper concluded that teachers participating 
in the initiative showed progress in incorporating 
formative assessment practices into classroom 
instruction. The white paper also issued recom- 
mendations on how the New York State Education 
Department could spread formative assessment 
training to other parts of the state. The district 
school board curriculum committee also re- 
quested a report from an external evaluator (it was 
not available at the time this report was written). 



Vermont Formative Assessment Project. The 
VT-FAP was launched as a pilot in 2006/07 and 
has become an ongoing initiative led by the Ver- 
mont Department of Education. 



Goals. As in New York, the Vermont initiative has 
a mix of program and implementation goals. The 
primary program goal has been higher student 
achievement through integration of formative 
assessment into regular classroom practice. 

There have been multiple implementation goals. 
One goal was to establish formal, school-based 
teacher learning communities (TECs) to promote 
school improvement across the state. TECs are an 
organizational structure to help teachers build col- 
lective responsibility for student learning and to 
support knowledge of formative assessment. Other 
implementation goals include spreading forma- 
tive assessment practices throughout the state 
and building state capacity to provide training in 
formative assessment to schools without support 
from paid vendors. 



Origins. VT-FAP grew 
out of the recognition by 
a new state commissioner 
of education and other 
state education lead- 
ers that schools wanted 
better guidance on how 



The primary program 
goal of VT-FAP has 
been higher student 
achievement through 
integration of formative 
assessment into regular 
classroom practice 
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During the pilot year, 
VT-FAP targeted 
teachers at all grade 
levels and content 
areas in schools with 
high concentrations 
of students from low- 
income households, 
but eligibility was 
later expanded to all 
interested schools 



to design local assessment plans, 
required by the state since 1997. 
Aware of the growing research on 
formative assessment, some state 
education agency staff recom- 
mended that classroom formative 
assessment serve as the foundation 
of a comprehensive system of local 
assessment that includes regular 
benchmark assessments as well as 
the annual state assessment. They 
believed that building teacher 
capacity for formative assessment 
is a core function that the Vermont Department of 
Education could help serve. The VT-FAP initiative 
emerged from these discussions. 



Leaders and roles. State education agency leaders 
from the Standards and Assessment Team played 
a primary role in conceptualizing, designing, and 
overseeing the initiative. They hired an external 
consultant to provide basic and train-the-trainer 
training in formative assessment and hired eight 
coaches as ongoing professional development 
facilitators and providers. They also helped to 
fund training and other initiative costs during the 
pilot year and have provided training and support 
throughout the life of the initiative. 



but eligibility was later expanded to all interested 

schools. 

• Targeted schools. State education agency leaders 
began with a small one-year pilot and invited 
schools receiving Title I school improvement 
funds to apply. Several non-Title I schools also 
applied and were accepted because the project 
budget could accommodate them. Nine schools 
participated in the pilot. Each participating 
school recruited teachers willing to commit to 
the entire two-year^ KLT professional develop- 
ment curriculum — an introductory, multiday 
summer institute and monthly TEC meetings. 
Following the pilot year, VT-FAP has been 
open to all interested schools. Starting in year 
3, the Vermont Department of Education also 
began to target schools designated in need of 
restructuring under accountability guidelines 
in the No Child Left Behind Act and schools 
applying for Mathematics and Science Partner- 
ship grant funding (see appendix C for more 
details). By spring of year 4 (2008), 55 schools 
were participating. 

• Targeted grade levels. The initiative was open 
to teachers of all grade levels in participating 
schools. 



Use of external partners. Following a call for 
proposals for a consultant to help the state train 
schools and teachers in formative assessment, 
the Vermont Department of Education selected 
Educational Testing Service (ETS) as the train- 
ing consultant and its Keeping Learning on Track 
(KLT) curriculum as the formative assessment 
professional development program. ETS staff 
provided introductory training to teacher groups, 
coaches, principals, and state education agency 
staff during the first summer of the pilot year, and 
they provided train-the-trainer training to coaches 
and state staff during spring of the pilot year. 

Targeted participants. During the pilot year, the 
initiative targeted teachers at all grade levels and 
content areas in schools with high concentra- 
tions of students from low-income households. 



• Targeted content area. The initiative was open 
to teachers of all content areas in participating 
schools. 

Funding. In the pilot year, the Vermont Depart- 
ment of Education paid for ETS services and the 
services of eight coaches through Title I, Title II-A, 
and GEAR UP federal funds. In subsequent years, 
participating schools were required to use local 
funds to cover summer institute training costs and 
the costs of KLT workbooks and training materials. 

Professional development. VT-FAP has provided 
consistent introductory and ongoing professional 
development in formative assessment to teachers 
since the pilot year. TEC facilitators, principals, 
district leaders, and state education agency officials 
have been offered the same form of introductory 
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training each year, but ongoing training to support 

the monthly TLC meetings has been less consistent. 

• For teachers. A mandatory two-day summer 
institute introduces teachers to the concepts 
and practices of formative assessment. During 
the pilot year, summer institute training was 
delivered by ETS staff, including a formative 
assessment expert who had helped write the KLT 
curriculum and had participated in drafting the 
CCSSO definition. In subsequent years, summer 
institute training sessions have been delivered by 
state education agency staff or one of the original 
eight coaches trained as summer institute train- 
ers. Teachers have received ongoing professional 
development in formative assessment through 
monthly TLC meetings that follow the content 
and structure of the KLT curriculum, and the 
guidance of a trained school-based TLC facilita- 
tor. Because KLT is a two-year curriculum, 
schools that have continued in VT-L AP beyond 
year 2 have had the option of continuing their 
own TLC meetings with their own curriculum. 

• For teacher leaders (TLC facilitators). Teach- 
ers and coaches in participating schools are 
chosen by their schools to receive TLC facilitator 
training, which consists of the same two-day 
summer institute training provided to teachers 
plus an additional day of specialized training. 
LTS provided the summer institute training for 
TLC facilitators during the pilot year, and state 
education agency staff and the eight original 
coaches (who served as TLC facilitators during 
the pilot year) provided training in subsequent 
years. These eight coaches also received ongoing 
support during the pilot year through manda- 
tory participation in regular online discussions 
with their coach peers and regular meetings 
with state staff. In subsequent years, none of the 
TLC facilitators has received formal ongoing 
support in formative assessment or TLC facilita- 
tion, although the state has set up online forums 
for peer support and collaboration. 

• For principals. Principals of participating 
schools have been required to attend the 



introductory summer training institute. Dur- 
ing the pilot year, principals were required to 
meet twice as a group with the state education 
agency, and in subsequent years principals of 
participating schools in restructuring have 
been required to meet monthly as a group 
with state officials for support and collabora- 
tion on implementation. Other participating 
principals have not received formal, on- 
going support in formative assessment or 
implementation. 

• For district leaders. Although participa- 
tion after the pilot year has been open to all 
schools and districts in the state, training 
and support have targeted schools and their 
staff rather than districts. The initiative has 
not provided specific training or support for 
district leaders. 



• For state education agency leaders. Some 25 
Vermont Department of Education officials 
participated in the initial summer training 
institute led by ETS, receiving the same two- 
day training delivered to teachers plus two ad- 
ditional days of training on TLC facilitation. 
In spring of the pilot year, these state staff and 
the eight coaches hired by the state received 
training from ETS as summer institute train- 
ers. This group of state officials and coaches 
has provided all subsequent summer institute 
training to new VT-L AP participants. State 
leaders have not received formal or ongoing 
support in formative assessment practices or 
KLT training since the pilot year, although 
they and coach trainers keep their training 
skills fresh by leading a summer institute ses- 
sion with other trainers each year. 



Formal evaluation. The 
Vermont Department of 
Education commissioned 
an external evaluation 
of the VT-LAP pilot year 
(Cole and O’Brian 2007). 
Based on participant 
feedback, the evaluation 



An external evaluation 
of the VT-FAP pilot year 
found that students 
and teachers had 
responded positively to 
the initiative and were 
in favor of continuing it 




10 



A DESCRIPTIVE ANALYSIS OF STATE-SUPPORTED FORMATIVE ASSESSMENT INITIATIVES IN NEW YORK AND VERMONT 



found that students and teachers had responded 
positively to the initiative and were in favor of con- 
tinuing it. These findings were instrumental in the 
state’s decision to continue the initiative. In year 4, 
the Vermont Department of Education commis- 
sioned a university survey on teachers’ experi- 
ences with VT-FAP implementation and levels of 
collaboration in schools undergoing restructuring. 
At the time of writing this report, results from this 
survey were not available. 



What strategies do state, district, and school leaders 
report using to support initiative implementation? 

Both NY-FATA and VT-FAP were launched to 
serve multiple program and implementation goals. 
Interviews with state, district, and school lead- 
ers identified several of the implementation goals 
noted in the literature review by Fixsen et al. (2005): 
getting teachers to adopt and integrate formative 
assessment practices into regular instruction, sup- 
porting and sustaining teacher use of formative as- 
sessment, and spreading or scaling use of formative 
assessment beyond pilot populations. The strategies 
used to achieve these three goals were classified into 
nine categories drawn from the implementation 
literature (Coburn 2003; Fixsen et al. 2005; Supovitz 
2008; table 1) and are discussed below. 



To rally support, leaders 
in New York and Vermont 
provided start-up 
training from well known 
formative assessment 
experts, and initiative 
leaders encouraged 
teachers to use formative 
assessment by creating 
environments in which 
teachers could feel 
comfortable exploring 
new practices 



1. Provide training from experts and credible evi- 
dence of benefits. To rally support, leaders in New 
York and Vermont provided start-up training from 

well known formative assessment 

experts. State and district leaders 
in New York, through collabora- 
tion with the NYCC and AACC, 
brought in a leading formative as- 
sessment scholar to provide foun- 
dational training to a pilot group of 
math coaches, school leaders, and 
other education officials during the 
first two summers of the initiative. 
Vermont leaders, working with 
ETS, brought in a developer of the 
KFT program and a recognized 
expert on formative assessment to 
introduce formative assessment 



strategies and TFCs to a pilot group of state of- 
ficials, school coaches, principals, and teachers 
during the summer of the VT-FAP pilot year. 

Three of the six interviewees in New York com- 
mented that having a known expert provide direct 
training lent greater credibility to the initiative 
and fostered buy-in by teachers and administra- 
tors. Similarly, three of the six interviewees in 
Vermont said that the strong reputation of the ETS 
expert promoted interest, trust, and participation 
in the VT-FAP initiative. A state and a district 
VT-FAP leader also suggested that the KFT 
teacher training curriculum, based on research 
that educators understood to be reputable and 
credible, helped teachers develop deeper engage- 
ment with formative assessment practices as they 
learned and worked with these practices over time. 

Four of the eight district and school NY-FATA and 
VT-FAP interviewees also noted that the initia- 
tives included extended forms of training to help 
teachers gather their own outcome evidence. Both 
initiatives established teacher training programs 
that encouraged teachers to try a small number 
of formative assessment strategies with their own 
students each month. Teachers observed how the 
strategies affected student performance, and then 
during the next month’s meeting they discussed 
the perceived successes and challenges of differ- 
ent strategies. Interviewees said that this approach 
allowed teachers to directly witness any student 
gains that emerged from formative assessment 
practices and that this was an important means of 
fostering and maintaining interest in the initiative. 

2. Create a safe environment for teachers to try out 
new practices. In both New York and Vermont, in- 
terviewees said that initiative leaders encouraged 
teachers to use formative assessment by creating 
environments in which teachers could feel com- 
fortable exploring new practices. Three interview- 
ees in New York said that math coaches provided 
constant support to teachers through coaching, 
modeling, and co-teaching methods of formative 
assessment. Math coaches clearly communicated 
that their work with teachers was not evaluative. 
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TABLE 1 

Implementation strategies used to promote adoption, sustainability, and spread of formative assessment in 
New York and Vermont, 2007-10 













Number of interviewees who 
identified the strategy 




Total 






Initiative goal 






State 


Stakeholder group 


of inter- 


Strategy 


Sustain- 
Adoption ability 


Spread 


New York Vermont 
(n = 6) (n = 6) 


State 
in = 4) 


District 
(n = 4) 


School 
(n = 4) 


viewees 
(n = 12) 


1. Provide training from experts and credible evidence of benefits 


Provide training from 
recognized experts 


✓ 






3 


3 


3 


2 


1 


6 


Provide research-based 
training curriculum 




✓ 






2 


1 


1 




2 


Show evidence of student 
gains 


✓ 


✓ 


✓ 


2 


2 




2 


2 


4 


2. Create a safe environment for 
teachers to try out new practices 


✓ 


✓ 




3 


3 


1 


2 


3 


6 


3. Align reforms with existing 
contexts and training efforts 


✓ 


✓ 


✓ 


4 


5 


3 


3 


3 


9 


4. Collaborate early with local 
leaders 


✓ 


✓ 




4 


3 


2 


4 


1 


7 


5. Employ both voluntary and 
mandatory participation 
approaches 


✓ 


✓ 


✓ 


3 


6 


2 


3 


4 


9 


6. Provide ongoing training and support for teachers and others at different levels of the system 


For teachers 




✓ 




3 


5 


2 


2 


4 


8 


For coaches/teacher leaders 




✓ 


✓ 


3 


4 


2 


3 


2 


7 


For principals 




✓ 




2 


2 


1 


1 


2 


4 


For district officials 




✓ 




1 






1 




1 


For state officials 




✓ 


✓ 


2 


2 


4 






4 


7. Establish accountability and 
monitoring methods for 
sustaining implementation 




✓ 




1 


4 


1 


2 


2 


5 


8. Build independent state and district capacity to sustain and spread teacher training 


Use outside experts to train 
state, district, or school 
coaches to become trainers 




✓ 


✓ 


4 


3 


3 


2 


2 


7 


Partner with local 
organizations to offer more 
teacher training 




✓ 


✓ 


3 


2 


3 


1 


1 


5 


9. Flarness enthusiasm at every level of the system to sustain and spread initiative, drawing on: 


Students 




✓ 


✓ 




2 




1 


1 


2 


Teachers 




✓ 


✓ 


2 


5 


2 


3 


2 


7 


Teacher leaders/coaches 




✓ 


✓ 


3 


2 


1 


4 




5 


Principals 




✓ 


✓ 


2 


3 




3 


2 


5 


District officials 




✓ 


✓ 


2 


1 




3 




3 


State officials 




✓ 


✓ 


2 


3 


4 




1 


5 



Source: Authors' analysis and tabulations of coded interview data. 
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A majority of 
interviewees in New 
York and Vermont 
described ways that 
state, district, and school 
initiative leaders aligned 
efforts to promote 
formative assessment 
with existing state and 
local priorities and 
other related education 
improvement activities 
to facilitate teacher 
adoption and ongoing 
use and to spread 
formative assessment 
practices and noted the 
importance of getting 
local administrators 
invested in the initiatives 



Similarly, three interviewees in 
Vermont described how VT-FAP 
leaders tried to encourage and 
sustain teacher use of formative 
assessment by creating training 
climates where it was acceptable 
to stumble. TLC meetings were 
designed as forums for teachers to 
freely share implementation suc- 
cesses and failures. At the school 
visited for this study, the princi- 
pal told teachers that they would 
not be evaluated while they were 
learning these new practices. She 
encouraged teachers to take risks 
and to fail visibly as long as they 
were learning to use formative as- 
sessment over time. 



3. Align reforms with existing 
contexts and training efforts. A 
majority of interviewees (4 of 6 
for NY-FATA and 5 of 6 for VT-FAP) described 
ways that state, district, and school initiative lead- 
ers aligned efforts to promote formative assess- 
ment with existing state and local priorities and 
other related education improvement activities to 
facilitate teacher adoption and ongoing use and 
to spread formative assessment practices. Two 
interviewees in New York noted that NY-FATA 
was directly linked to the district’s efforts to 
increase student progress monitoring. Two other 
interviewees said that NY-FATA training activi- 
ties complemented efforts to support the pilot 
district’s new math curriculum. An interviewee 
in Vermont explained that VT-FAP was created 
to help schools and districts establish one piece 
of the state’s broader vision of a comprehensive 
system of assessment. Two other Vermont inter- 
viewees remarked that VT-FAP complemented the 
school’s earlier efforts to train teachers in develop- 
ing student learning goals, adopting a schoolwide 
behavioral intervention program, and teaching 
a new math curriculum. A New York district 
interviewee and a Vermont school interviewee also 
noted that staff had already been primed before 
the initiative to engage in collaborative training: 



New York teachers and math coaches in the pilot 
district had begun to adopt a collaborative coach- 
ing model before the pilot, and teachers in the 
Vermont school were already meeting in house 
teams before VT-FAP brought more structured 
TLCs to the school. 

Initiative leaders in New York and Vermont also 
sought to scale up the initiatives by linking them 
to other grant and school improvement programs. 
New York district leaders hoped to spread forma- 
tive assessment practices to other schools in the 
pilot district by applying for another round of 
Mathematics and Science Partnership funding. 
State leaders in Vermont hoped to spread the initia- 
tive to other schools and districts by requiring ap- 
plicants for Mathematics and Science Partnership 
grants to implement VT-FAP. Vermont state lead- 
ers took the further step of requiring all schools 
designated as in need of restructuring under No 
Child Left Behind Act accountability guidelines 
to adopt either VT-FAP or a program establish- 
ing critical friends groups.^ Both programs aim to 
transform teacher professional development and 
practice by instituting school-based TLCs. State 
leaders therefore worked to scale up VT-FAP by 
linking it to the state’s menu of options for support- 
ing schools deemed in need of major intervention. 

4. Collaborate early with local leadership. New 
York and Vermont interviewees at the state and 
district levels noted the importance of getting 
local administrators invested in the initiatives so 
that they in turn would help engage teachers and 
sustain their interest in using formative assess- 
ment. A New York interviewee indicated that the 
state developed an early partnership with pilot 
district leaders, whose active involvement in 
NY-FATA made the initiative possible. Both New 
York district interviewees said that they launched 
NY-FATA by speaking first with principals and 
then by selecting schools whose principals volun- 
teered to join the program. One of the New York 
school interviewees confirmed that district leaders 
had many conversations with the school princi- 
pal to help determine whether NY-FATA was a 
good fit for the school. She said that the district’s 
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collaborative approach ultimately helped secure 
the principal’s cooperation. 

The Vermont Department of Education took a 
similar approach in launching VT-FAP. One 
interviewee described how state leaders held a 
community discussion explaining the initiative to 
a targeted set of principals and superintendents 
overseeing Title I schools. Schools were chosen for 
the VT-FAP pilot based on principals’ applications 
and their commitment to securing teachers’ par- 
ticipation in the initiative. Two interviewees noted 
that principals were required to attend the initial 
VT-FAP summer institute training to ensure that 
they understood formative assessment and how to 
support teachers’ efforts to learn to use it through 
TFCs and ongoing classroom experience. Another 
interviewee indicated that at the school visited for 
this study, school leaders worked hard from the 
beginning to secure the local school board’s back- 
ing for the VT-FAP initiative to ensure that the 
community would support the school’s formative 
assessment activities. 

5. Employ both voluntary and mandatory partici- 
pation approaches. New York district leaders and 
Vermont state leaders originally solicited school 
participation in their formative assessment initia- 
tives by appealing to school principals to volunteer 
their schools for the initiative. Once an initial 
group of principals had committed their schools 
to the initiative, state, district, and school leaders 
used a mix of voluntary and mandatory approaches 
to secure teacher participation and support. A New 
York district interviewee and a school interviewee 
noted that participation during the initiative’s first 
year was mandatory for grade 4 and 5 teachers in 
schools whose principals had volunteered for the 
initiative. However, as one interviewee explained, 
strong union rules mean that requirements do 
not necessarily translate into teacher compliance. 
To gain teacher buy-in, a school interviewee said 
that the principal heavily recruited a few teachers, 
targeting those she believed would be enthusiastic 
and would attract other teachers. The other school 
interviewee explained that participation became 
fully voluntary in subsequent years. 



A similar mix of incen- 
tives and requirements 
characterized teacher 
recruitment in Vermont. 

In the school visited for 
this study, a school inter- 
viewee and a district in- 
terviewee said that school 
leaders initially solicited 
teacher volunteers for 
VT-FAP training. At the 
same time, these school 
leaders heavily recruited 
a few influential teachers 
they believed would be 
receptive to the initiative 
and could help persuade 
others to participate. 

Three interviewees ex- 
plained that the primary 
approach for spreading 

VT-FAP across the state was to start with small 
groups of volunteers and to allow the program to 
grow organically through positive word of mouth. 



New York district leaders 
and Vermont state 
leaders used a mix of 
voluntary and mandatory 
approaches to secure 
teacher participation and 
support, and to sustain 
teachers' efforts to learn 
and deepen skills in 
formative assessment, 
both NY-FATAand 
VT-FAP established 
ongoing professional 
development and 
provided ongoing training 
and support for school, 
district, and state officials 
to build capacity to 
support teacher training 



At the same time, there were also mandatory pres- 
sures for participation. One district interviewee 
said that many of the principals who volunteered 
for VT-FAP were in charge of underperforming 
schools that were required by the state commis- 
sioner of education to adopt school improvement 
measures. Teachers, teacher leaders, and princi- 
pals that volunteered for VT-FAP had to attend an 
initial summer training session and participate in 
TFCs for at least two years. One interviewee noted 
that during the pilot year, TFC leaders had to par- 
ticipate in regular online community meetings. At 
the school visited for this study, the principal had 
made VT-FAP training and participation in TFCs 
mandatory for all teachers by the third year of the 
initiative, to spread formative assessment practices 
throughout the school. Also in the third year, state 
officials worked to spread VT-FAP across the state 
by making participation a requirement for Math- 
ematics and Science Partnership grants and one 
of two mandatory intervention options for schools 
classified as in need of restructuring. 
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6. Provide ongoing training and support for teachers 
and others at different levels of the system. To sus- 
tain teachers’ efforts to learn and deepen skills in 
formative assessment, both NY-FATA and VT-FAP 
established ongoing professional development. 

Both initiatives also provided ongoing training and 
support for school, district, and state officials to 
build capacity to support teacher training. 



Teachers in the pilot district in New York re- 
ceived ongoing training in formative assessment 
from math coaches through monthly workshops 
(year 1) and then through voluntary individual 
or small-group coaching sessions (year 2). 

Both school interviewees in New York said that 
teachers who requested coaching in formative 
assessment received regular, highly individual- 
ized support from the math coach. Vermont 
teachers participating in VT-FAP also received 
ongoing professional development in formative 
assessment, but from teacher peers in monthly 
TLC meetings, the written KLT curriculum, and 
a trained TLC facilitator. At the school visited 
for this study, two interviewees noted that the 
school’s professional development coordina- 
tor, the principal, and two school-based teacher 
coaches trained as TLC leaders provided addi- 
tional support to individual teachers on request as 
part of their regular instructional support duties. 



Initiative leaders took 
different approaches 
in the two states 
to accountability 
and monitoring for 
sustaining initiative 
implementation, but 
leaders in both states 
worked to ensure that 
the formative assessment 
initiatives would be 
sustainable and scalable 
within their jurisdictions 
without the ongoing 
involvement of their 
original external partners 



Teacher trainers received dif- 
ferent levels of ongoing support 
from each initiative. Math coaches 
in New York received formal, 
ongoing training in formative 
assessment and teacher coaching 
from the NYCC during the first 
two years. In Vermont, accord- 
ing to two interviewees, the state 
education agency provided regular 
support to TLC facilitators during 
the pilot year through meetings 
and an online community forum. 
After the pilot year, the Vermont 
Department of Education held no 
more face-to-face meetings with 
TLC facilitators, and participation 



in the online community forum was low. One 
interviewee said that some educational services 
agencies had convened mid-year meetings to 
provide support for trained TLC leaders but that 
others had not. 

Both initiatives provided ongoing support to par- 
ticipating principals, but only NY-FATA provided 
ongoing support to district and state officials. As 
reported by two New York interviewees, district 
leaders and the NYCC provided information about 
the initiative and advice on implementation to 
principals during regular group meetings. Two 
Vermont interviewees said that the state pro- 
vided support for principals of the original nine 
pilot schools and of schools under restructuring 
through regular principal-community meetings. 
Three interviewees noted that the NYCC also 
continued to support district and state leaders 
with planning and future scaling strategies during 
the first two years. In contrast, two interviewees 
in Vermont said that the state had been overseeing 
the development and growth of VT-FAP without 
ETS support since the end of the pilot year. 

7. Establish accountability and monitoring methods 
for sustaining implementation. Initiative leaders 
took different approaches in the two states to ac- 
countability and monitoring for sustaining initia- 
tive implementation. New York interviewees did 
not describe any direct methods for ensuring that 
teachers were using formative assessment prac- 
tices in their classrooms. Rather, one interviewee 
said the district is subject to numerous audits and 
requests for data on various district programs 
from the New York State Education Department. 
She did not mention any regular data requests 
other than annual federal Mathematics and Sci- 
ence Partnership grant reporting requirements on 
overall progress of NY-FATA implementation. 

State, district, and school VT-FAP leaders described 
using more direct methods to monitor formative 
assessment practices in the classroom. Two inter- 
viewees indicated that the Vermont Department 
of Education actively monitored multiple groups 
participating in the initiative. For the nine pilot 
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schools and schools under restructuring, the state 
held principals accountable for supporting forma- 
tive assessment through monthly meetings at which 
principals discussed implementation issues. The 
state did not monitor TLCs directly but contracted a 
university to conduct a survey of teaching condi- 
tions in schools to learn more about how TLCs were 
functioning. The state tracked student performance 
by requiring schools under restructuring to publicly 
report student results on state and local assessments 
and to submit progress monitoring charts and a 
support plan for each student who was not reaching 
standards. Three interviewees in Vermont said that 
at the school visited for this study, the principal for- 
mally held teachers accountable for using formative 
assessment strategies in their classrooms and that 
TLC leaders and teachers informally held each other 
accountable for executing the strategies presented in 
their monthly TLC action plans. 

8. Build independent state and district capacity to 
sustain and spread teacher training. One interviewee 
in New York and two in Vermont stated that they 
worked to ensure that the formative assessment 
initiatives would be sustainable and scalable within 
their jurisdictions without the ongoing involvement 
of their original external partners. During the three 
years of NY-FATA, New York district leaders turned 
to expert consultants at the NYCC and the AACC to 
get as much training and support for math coaches 
as possible, so that they in turn could support and 
expand the number of teachers skilled in forma- 
tive assessment. Both New York school interview- 
ees noted that some math coaches began to take 
ownership of teacher training in the second year, as 
several coaches worked together to create an intro- 
ductory 30 -hour professional development course. 
Formative Assessment 1. This course was offered 
to teachers in their own schools in year 2 and to all 
teachers in the district in year 3. One math coach 
created a follow-up course. Formative Assessment 2, 
and offered it to teachers across the district in year 
3. One of the two district interviewees also reported 
that math coaches met monthly with a group of 
districtwide professional development leaders 
to spread formative assessment knowledge and 
practices. The aim was to develop a broad group of 



teacher coaches who could 
spread formative assess- 
ment practices throughout 
district schools. 



Five of the six New York 
interviewees and all six 
Vermont interviewees 
reported that initiative 
participants helped 
sustain and spread 
the initiative through 
their enthusiastic 
dedication to formative 



assessment practices 
and their strong 
advocacy expressed in 
formal presentations 
to other educators 
in their districts and 
across the state 



After VT-FAP’s pilot 
year, state education 
leaders took steps to 
reduce dependence on 
ETS consultants. Two 
interviewees explained 
that state education lead- 
ers asked ETS experts 
to train a group of state 
officials and the original 
eight external coaches to 
become KLT trainers in 
spring of the pilot year. 

In subsequent years, this group of KLT trainers led 
the VT-FAP summer training institutes for teach- 
ers, TLC leaders/facilitators, and administrators 
from across the state who were joining the initia- 
tive. According to one interviewee, Vermont state 
leaders also recognized a concern that New York 
state and district initiative leaders did not discuss: 
the future need for new KLT trainers because of 
turnover and attrition in the original corps of 
trainers. The Vermont Department of Education 
discussed plans with ETS to teach state education 
agency staff to train new KLT trainers. 



Both initiatives also partnered with local organi- 
zations to spread formative assessment practices 
beyond the original participating schools. Three 
interviewees in New York noted that the pilot district 
worked with the district’s Teacher Center to offer the 
formative Assessment 1 and 2 courses. Similarly, 
two interviewees in Vermont explained that the state 
worked with regional educational service agencies to 
get logistical support in running VT-FAP summer 
training institutes throughout the state. 



9. Harness enthusiasm at every level of the system 
to sustain and spread formative assessment. Five of 
the six New York interviewees and all six Vermont 
interviewees reported that initiative participants 
helped sustain and spread the initiative through 
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Future research will need 
to examine whether the 
strategies employed by 
state, district, and school 
leaders to promote the 
sustainability of each 
initiative can withstand 
multiple and potentially 
competing goals and 
the extent to which 
leadership by a district 
or a state education 
agency affects scalability 



their enthusiastic dedication to 
formative assessment practices 
and their strong advocacy ex- 
pressed in formal presentations to 
other educators in their districts 
and across the state. A New York 
interviewee said that teachers in 
the pilot district organized cross- 
school meetings to observe and 
provide feedback on one another’s 
classroom strategies, thus help- 
ing support and sustain formative 
assessment. Another New York 
interviewee said that teachers 
promoted NY-FATA by describ- 
ing their experiences at statewide conferences. 

Five Vermont interviewees said that by launching 
VT-FAP with volunteer teachers, who were more 
likely to actively engage in training, the initia- 
tive established a core group of teachers whose 
enthusiasm helped persuade others to participate. 
Two Vermont interviewees also remarked that stu- 
dents helped draw teachers to the initiative, since 
students exposed to formative assessment in lower 
grades came to expect it in higher grades as well. 



Three New York interviewees and two Vermont in- 
terviewees commented on how the work ethic and 
commitment of teacher leaders (including math 
coaches in New York and school administrators in 
Vermont) helped motivate teachers for formative 
assessment training and attracted new teachers 
to the initiatives. Two New York interviewees also 
noted that principals of participating schools, 
district leaders, and state officials helped spread 
the word about NY-FATA by speaking about their 
implementation experiences at state conferences. 
Similarly, three Vermont interviewees noted how 
principals’ enthusiasm for VT-FAP helped con- 
vince teachers to participate. One interviewee 
remarked that educational services agencies were 
helping spread the word about formative assess- 
ment by advertising VT-FAP training institutes 
throughout the state, and two interviewees de- 
scribed the active role of the Vermont Department 
of Education in promoting the initiative. For exam- 
ple, state agency staff gave presentations around the 



state about VT-FAP, distributed buttons that read 
“What’s formative about it?” to remind teachers to 
think about using formative assessment practices, 
and assigned at least one staff person in the agency 
to monitor, oversee, and support the initiative. 



DISCUSSION 

To provide additional insights into the structure 
and operation of the NY-FATA and VT-FAP 
initiatives, this section highlights similarities and 
differences in the primary components of each 
initiative and in the implementation strategies 
identified from the literature and the analysis of 
interview data. 



Comparison of the two initiatives 

Goals. Both NY-FATA and VT-FAP had multiple 
goals. Their program goals were similar: to improve 
student achievement through teacher use of forma- 
tive assessment practices as part of daily instruc- 
tion. So were their broad implementation goals: to 
successfully install, sustain, and spread formative 
assessment practices beyond pilot sites. Each initia- 
tive also had other implementation goals that were 
sometimes tangential to teacher use of formative 
assessment and that differed from one another. 
NY-FATA aimed to help the district achieve day-to- 
day student progress monitoring, to improve teach- 
ers’ math content knowledge and pedagogical skills, 
and to help the New York State Education Depart- 
ment determine whether an initiative focused on 
building teacher capacity in formative assessment 
was an effective school improvement model for 
other districts. VT-FAP was designed not only to 
build teacher capacity in formative assessment but 
also to establish TECs as a means of reorganizing 
schools and reshaping teachers’ professional learn- 
ing practices across the state. 

Multiple program goals raise questions about 
whether the goals are fully compatible and 
whether some goals receive greater priority. 

The possibility of shifting priorities could have 
implications for teacher training in formative 
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assessment. For example, a greater desire within 
the New York pilot district to focus on teachers’ 
math content knowledge could divert energy from 
building teacher skills in formative assessment. 

Or the heightened interest in establishing TLCs 
in Vermont could turn education leaders’ focus 
toward building these structures without a strong 
formative assessment curriculum component. 
Future research will need to examine whether the 
strategies employed by state, district, and school 
leaders to promote the sustainability of each 
initiative can withstand multiple and potentially 
competing goals. 

Origins and leaders. NY-FATA and VT-FAP had 
different origins and different initiative leadership 
structures and roles. NY-FATA emerged from an 
individual district’s request and formal applica- 
tion to the New York State Education Department 
for support in developing district capacity to train 
teachers in formative assessment. The pilot district 
thus became the NY-FATA leader. The initia- 
tive enabled the district to work with the NYCC 
and A ACC to develop a program — almost from 
scratch — to train teachers in formative assessment. 
The state education agency played a supporting role 
by funding NY-FATA and bringing the services of 
the NYCC and AACC to the district. In contrast, 
VT-FAP emerged in large part from the Vermont 
Department of Education’s desire to help schools 
across the state adopt formative assessment. The 
state became the leader of VT-EAP, choosing an 
existing professional development program in 
formative assessment (KLT) and working to seed, 
sustain, and spread the program statewide. 

The different origins and leaders associated with 
each initiative could have implications for scalabil- 
ity, as the example of NY-EATA and VT-FAP show. 
Further research is needed, however, on the extent 
to which leadership by a district or a state education 
agency affects scalability. NY-FATA, with a pilot 
district as its leader, focused primarily on building 
the pilot district’s capacity to support teacher use of 
formative assessment and to spread its use across 
the district. It placed less focus on how to build 
such capacity in other districts. The New York State 



Education Department 
was interested in the scal- 
ability of NY-FATA state- 
wide, but as an observer 
of the initiative’s develop- 
ment and implementation, 
it did not actively work 
to spread the initiative to 
other districts. In contrast, 

VT-FAP was launched 

and led by the Vermont Department of Education, 
which was interested from the start in a professional 
development program that could readily be brought 
to scale statewide. As a result, VT-EAP has spread 
to schools throughout Vermont. 



Both NY-FATA and 
VT-FAP relied on 
external partners for 
support, but the two 
initiatives targeted 
different participants 
in different ways and 
with different costs 



Use of external partners. Both NY-FATA and VT- 
FAP relied on external partners for support, but in 
different ways and with different costs. Through 
the New York State Education Department, the 
pilot district in New York drew on the direct and 
free assistance of the NYCC and AACC for almost 
the full three years of the initiative. The Vermont 
Department of Education contracted ETS to use its 
KLT program, to provide initial summer training 
in the program, and to train trainers during the 
pilot year. In subsequent years, Vermont Depart- 
ment of Education staff and other in-state trainers 
provided the summer training for new partici- 
pants, while the KLT curriculum — implemented 
through school-based TLCs — has provided the 
structure and content of VT-EAP’s ongoing profes- 
sional development program. 



These differences in the roles of external partners 
contribute to different challenges in bringing 
programs to scale. Because math coaches in the 
New York pilot district learned to train teachers 
in formative assessment, math coaches can help 
spread NY-FATA to other district schools. But 
because the NYCC and AACC played fundamental 
roles over several years in developing district math 
coaches’ training skills, NY-FATA may be difficult 
to replicate in other districts without the assistance 
of external groups that can help build a corps of 
district-based trainers. The role of ETS, the VT-EAP 
external partner, was limited to helping the state 
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develop an in-state corps of summer institute train- 
ers during the pilot year, thus enabling the state 
to scale up VT-FAP independently in subsequent 
years. Initiative leaders in Vermont now face a scal- 
ing challenge that leaders in New York have not yet 
confronted: finding a way to replenish and renew its 
corps of in-state trainers as the original corps loses 
members and as trainers’ skills become stale. At the 
time of this report, Vermont was considering rehir- 
ing ETS to help address this issue. 

Targeted participants. The two initiatives targeted 
different participants. NY-FATA focused on 
building the capacity of grade 4 and 5 teachers in 
formative assessment for math instruction during 
the first year, and participation was expanded 
to teachers in other grades in subsequent years. 
VT-FAP had no specific grade or content area 
focus. New York state leaders worked with a single 
pilot district while Vermont state leaders, like 
others in a recent national formative assessment 
initiative studied by Weinbaum (2009), worked 
directly with individual schools rather than with 
districts. The choice among state education leaders 
to work directly with districts or schools could 
have different implications for initiative adoption, 
sustainability, and scale, but the implications are 
unclear and require further research. 



Future research could 
examine whether 
extended train-the- 
trainer efforts delivered 
by outside experts 
produce stronger 
trainers than one-time 
train-the-trainer efforts 



Funding. Different funding sources have different 
implications for the sustainability and scalability of 
each initiative. NY-FATA was funded by a three- 
year federal Mathematics and Science Partnership 
grant administered by the state; when grant fund- 
ing ended so did the initiative. The district hopes to 
spread formative practices to other schools within 
the district through its corps of trained math 

coaches, and it hopes to strengthen 
scaling activities by applying for 
another round of Mathematics and 
Science Partnership funding. The 
Vermont Department of Education 
funded the pilot year of VT-FAP 
drawing on several sources of 
federal funds. In subsequent years, 
schools have paid their own way. 

If the New York district continues 



to rely on Mathematics and Science Partnership 
funding to scale NY-FATA, that could restrict the 
district’s ability to spread formative assessment 
practices to subjects beyond math and science. The 
Vermont Department of Education may face a sim- 
ilar issue for some local sites as it makes VT-FAP 
participation a requirement for Mathematics and 
Science Partnership grant applicants in Vermont. 

Professional development. Both initiatives pro- 
vided initial and ongoing professional devel- 
opment to teachers in formative assessment, 
although training came from different sources. 
Teachers participating in NY-FATA never re- 
ceived direct training from NYCC and AACC staff, 
whereas teachers who participated in VT-FAP’s 
pilot year summer training institute received 
direct professional development from ETS experts. 
Supovitz (2008) has argued that teacher training 
in school reform efforts may be more effective 
when the training is delivered by knowledgeable 
experts who are deeply familiar with reform prac- 
tices, rather than by trained trainers. It is unclear 
whether the quality of training that Vermont 
teachers received from ETS during the first year 
was different from the quality of training that 
teachers received from in-state trainers in subse- 
quent years. Similarly, it is unclear how the quality 
of training that NY-FATA math coaches received 
directly from NYCC staff may have differed from 
the quality that teachers received from the math 
coaches. The NYCC provided direct, formal, and 
ongoing support to math coaches for the first two 
years of NY-FATA, whereas ETS provided only a 
one-time training session for state staff and eight 
coaches to become summer institute trainers in 
Vermont. Future research could examine whether 
extended train-the-trainer efforts delivered by 
outside experts produce stronger trainers than 
one-time train-the-trainer efforts. 

Formal evaluation. A qualitative review of VT- 
FAP’s implementation and outcomes during the 
pilot year helped the Vermont Department of 
Education decide to continue the initiative and 
to think strategically about scaling. Reports of 
NY-FATA implementation and outcomes after 
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years 1 and 2 were presented to state and district 
staff, but interviewees did not articulate any clear 
plans emerging from the findings. 



Implementation strategies 

NY-FATA and VT-FAP both pursued strategies 
that have been suggested in the literature for 
promoting program adoption, sustainability, and 
spread. First, they engaged in careful site selec- 
tion (see Fixsen et al. 2005; McLaughlin and Mitra 
2001; Supovitz 2008). They began site selection by 
holding preliminary conversations with princi- 
pals, assessing their levels of interest, and solicit- 
ing their participation. Vermont went through a 
formal site selection process during the pilot year, 
asking schools to demonstrate their capacity and 
commitment in written applications. 

Second, both initiatives pursued strategies sug- 
gested by researchers such as Coburn (2003) and 
McLaughlin and Mitra (2001) to align goals and ac- 
tivities with existing policies and school improve- 
ment efforts. Interviewees indicated that these site 
selection and alignment strategies were intended 
to promote teacher adoption, sustained use, and 
spread of formative assessment practices, but 
further research is required to determine whether 
these strategies helped achieve these outcomes. 

Third, both initiatives considered whether to work 
with existing or new communication structures 
within schools and districts to seed and spread the 
use of formative assessment among teachers. Re- 
search has suggested that such decisions may affect 
how well new practices diffuse through organiza- 
tions (Supovitz 2008). In New York, the pilot district 
chose to have the NYCC transfer knowledge and 
skills in classroom formative assessment to teachers 
through an existing group of math coaches. Further 
research could investigate whether formative as- 
sessment training within the pilot district is more 
effective when delivered by math coaches in special 
monthly group meetings (used in year 1), as part 
of regular coaching activities (used in year 2), or in 
formats involving other trainers. VT-FAP provided 
ongoing teacher training in formative assessment 



through dedicated 
monthly TLC meetings. 
One goal of the initiative 
has been to create TLCs 
in schools throughout the 
state. Further research 
could investigate whether 
formative assessment 
practices spread more 
quickly among teach- 
ers through TLCs or 
through existing school 
professional development 
arrangements. 



Interviewees in both 
states described 
strong efforts to create 
safe and supportive 
school cultures to ease 
the challenges that 
arise when trying to 
change fundamental 
teaching practices and 
to encourage as much 
voluntary participation 
as possible to promote 
initiative buy-in 



Fourth, NY-FATA and VT-FAP provided ongoing 
teacher professional development and built teacher 
support structures, as recommended by research- 
ers (Coburn 2003; Elmore 2004; and Supovitz 
2008), to achieve deep and sustained use of forma- 
tive assessment. Both state education agencies 
also collaborated early with local school or district 
leaders to ensure local ownership and ongoing 
administrator support, as recommended in the 
literature (McLaughlin and Mitra 2001; Coburn 
2003; Elmore 2004; and Supovitz 2008). A strategy 
recommended by Coburn (2003) that NY-FATA 
and VT-FAP leaders did not appear to pursue 
was to build capacity within local sites to create 
future streams of funding to spread each initia- 
tive. District leaders of NY-FATA planned to apply 
for another round of Mathematics and Science 
Partnership funding and to use regular district 
funds to spread formative assessment practices to 
more teachers and schools using math coaches. 
State leaders of VT-FAP have asked schools and 
districts to pay their own expenses after the pilot 
year. These approaches offer opportunities to scale 
up initiatives, but they depend on grant availabil- 
ity and school and district funds. 



Some of the strategies that initiative leaders 
described for pursuing common implementa- 
tion goals expand on strategies described in the 
literature. For example, interviewees in both states 
emphasized the value of using expert trainers and 
credible evidence to promote initiative adoption. 
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sustainability, and spread. Interviewees in both 
states also described strong efforts to create safe 
and supportive school cultures to ease the chal- 
lenges that arise when trying to change funda- 
mental teaching practices. Leaders of both initia- 
tives said that they tried to encourage as much 
voluntary participation as possible to promote 
initiative buy-in, and they partnered with local 
organizations (the Teacher Center in New York’s 
pilot district and educational services agencies in 
Vermont) to huild local scale-up capacity. How ef- 
fective these strategies are in achieving implemen- 
tation goals is a question for future research. 



Limitations and contributions of this study 

The study team interviewed only a small number 
of respondents and gathered data from only one 
participating district and one school for each 
initiative recommended by initiative leaders as 
sites that had displayed a strong capacity to work 
with the state or where the initiative had been 
implemented most fully. Such sites are unlikely 
to be typical of sites participating in each initia- 
tive. Furthermore, the knowledge and opinions 



of the two state leaders and the two district and 
school leaders interviewed in each state might 
not represent those of other initiative leaders and 
participants. This study does not provide evidence 
of the effectiveness of any of these initiatives, 
their implementation efforts, or the roles of state 
education agencies in school, teacher, or student 
outcomes. 

This study’s main contribution is the detailed 
descriptions of two state-supported initiatives 
that promote large-scale teacher use of formative 
assessment practices as defined by the CCSSO, as 
well as exploratory ideas for future research and 
practice. For many teachers, incorporating such 
practices into daily instruction requires funda- 
mental changes in their approaches to teaching 
and learning. Researchers have noted the difficulty 
of achieving such changes, and this study chron- 
icles some of the steps that two initiatives have 
taken to ease this challenge. Although descriptions 
of implementation strategies rest on the reports of 
only a handful of interviewees, they were chosen 
as likely to be the most knowledgeable informants 
for each initiative. 
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APPENDIX A 
STUDY METHODS 

This research study posed two research questions: 

• In Northeast and Islands Region states where 
state education agencies are supporting 
formative assessment initiatives, what are the 
primary components of each initiative? 

• What strategies do state, district, and school 
leaders report using to support implementa- 
tion of formative assessment initiatives? 

For the first research question, the primary com- 
ponents examined were: 

• Goals. Purposes of the initiative from state 
and local perspectives, including program and 
implementation goals. 

• Origins. Description of how the initiative started. 

• Leaders and roles. In-state public education 
agencies that helped lead or support the initia- 
tive and the roles they played. 

• Use of external partners. Roles of external 
partners. 



• Targeted participants. Types of schools, grade 
levels, and teacher content areas targeted for 
inclusion in the initiative. 

• Funding. State-administered federal grants 
and local resources supporting the initiative. 

• Professional development. Types of teacher 
professional development and training offered 
to school, district, and state leaders to support 
initiative implementation. 

• Formal evaluations. Data collected and 
reports generated to assess initiative progress 
and impacts. 

For the second research question, the study team 
interviewed two state, two district, and two school 
leaders involved in each initiative to gain their 
perspectives on the strategies they pursued to 
achieve the initiative implementation goals of 
adoption, sustainability, and spread. 



Project sample selection 

Purposive (nonrandom) sampling was used to 
select state-supported formative assessment initia- 
tives in the Northeast and Islands Region, based 
on specific criteria (O’Leary 2004; table Al). 



TABLE Al 

Selection criteria for Northeast and Islands Region state formative assessment initiatives, 2009-10 



Criterion 


Rationale 


1. 


identified by the state or an outside organization as 
a coordinated effort to build capacity in formative 
assessment within the state 


To limit the scope of this project to initiatives with official 
state recognition and to provide state education leaders with 
information about formal efforts to which states have devoted 
staff or other resources 


2. 


Promoted formative assessment practices consistent 
with the definition of formative assessment established 
by the Council of Chief State School Officers 


To minimize confusion over the concept of formative assessment 
and limit the scope of the project to initiatives promoting the 
same set of changes in teaching and learning practices 


3. 


Supported by the state 


To examine how state education agencies have helped districts 
and schools build capacity in formative assessment 


4. 


Unique to the state and not part of a larger national 
initiative or evaluation 


To provide examples of initiatives that jurisdictions in the region 
are able to implement at the state level 


5. 


Implemented in at least one district within the state for 
at least one year 


To limit data collection to programs with a record of activity that 
state and local educators can describe and critique 



Source.- Authors. 
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To identify initiatives, the study team searched 
websites linked to state departments of educa- 
tion and online publications provided by regional 
and national organizations involved with student 
assessment, such as the New York Comprehensive 
Center (NYCC) and the Council of Chief State 
School Officers (CCSSO), using the search term 
“formative assessment.” Two state-supported ini- 
tiatives were identified that met the study criteria: 
the New York Formative Assessment Technical 
Assistance Study (NY-FATA) and the Vermont 
Formative Assessment Project (VT-FAP). Public 
documents describing the practices promoted by 
these two initiatives helped confirm that both were 



using the CCSSO’s definition of formative assess- 
ment or a consistent definition (see table B1 in 
appendix B). Other formative assessment activities 
in the Northeast and Island Region did not meet 
the project’s selection criteria (see box Al). 

Data on the primary components of each initiative 
were collected from sources within each state edu- 
cation agency. To develop a more detailed under- 
standing of how each initiative was implemented in 
specific settings, data were collected from one dis- 
trict and school participating in each initiative. The 
districts and schools were selected based on recom- 
mendations by state and district leaders of each 



BOX A1 

Examples of initiatives excluded 
from this study 

Public documents suggested that sev- 
eral state-supported initiatives fit the 
study’s selection criteria, but further 
examination ruled them out: 

• The Connecticut Grade 3-5 For- 
mative Assessment Mathematics 
Pilot Project began in 2006 with 
aspirations to support formative 
assessment practices in Con- 
necticut classrooms but evolved 
into a benchmark assessment 
program and was therefore not 
examined in this report (did not 
meet criterion 2 in table Al). 

• The Connecticut Accountability 
for Learning Initiative includes 
training for teachers in collab- 
oratively developing, administer- 
ing, and scoring instructional 
responses to group-administered 
classroom assessments. The 
program calls this work “com- 
mon formative assessment,” a 
type of activity that differs from 
the Council of Chief State School 



Officers (CCSSO) definition of 
formative assessment (did not 
meet criterion 2). 

Public documentation describ- 
ing the Massachusetts Galileo 
Pilot Project suggested that it has 
a formative assessment com- 
ponent, but closer examination 
found the focus to be on provid- 
ing a computer-based bench- 
mark assessment system to Mas- 
sachusetts districts. Although 
the Galileo benchmark system 
has the capability to support 
teacher development and use of 
common formative assessments, 
the project has not been used to 
support the CCSSO definition of 
formative assessment (did not 
meet criterion 2). 

A search of the New York State 
Education Department website 
revealed that the department’s Of- 
fice of Vocational and Educational 
Services for Individuals with 
Disabilities had issued a series of 
teacher improvement resource 
guides in 2008 that discuss for- 
mative assessment. But the guides 



are references rather than a coor- 
dinated effort to build capacity in 
formative assessment within the 
state (did not meet criterion 1). 

• The Rhode Island Department 
of Education has a page on its 
website that describes, defines, 
and encourages formative assess- 
ment practices, but the webpage 
simply provides information and 
is not evidence of an organized 
state effort supporting forma- 
tive assessment (did not meet 
criterion 1). 

• Erom April 2007 to June 2008, 
Connecticut and Maine par- 
ticipated in a 10-state initiative 
and evaluation supported by 
the CCSSO and funded by the 
U.S. Department of Education 
to improve assessment practices 
in low-performing high schools. 
The efforts under this initiative 
do not focus on initiatives that 
states have implemented within 
their jurisdiction without the 
infrastructure or support of a 
national network of peer collabo- 
rators (did not meet criterion 4). 



APPENDIX A. STUDY METHODS 



23 



initiative. NY-FATA was implemented in only one 
district in New York. District leaders were asked 
to nominate and rank schools for study inclusion, 
focusing on schools that had implemented the ini- 
tiative most fully. The most frequently and highly 
recommended school was invited to participate, 
and its school leaders agreed. In Vermont, the state 
education agency worked primarily with schools 
rather than districts to implement the initiative. 
State initiative leaders were asked to nominate and 
rank schools that had implemented the initiative 
most fully. The most highly recommended school 
was invited to participate, and its leaders agreed. 
Table A2 provides background data on the schools 
and districts examined in this study. 



Data sources 

The project drew on two data sources to address 
the research questions: 



Publicly available information. To describe the 
primary components of each state initiative, 
the study team consulted public documents 
and information available online, as well as 
material from initiative leaders and partners. 
Online sources included state education 
agency websites, online education research 
databases, and vendor or partner organiza- 
tion websites. Information and documents 
included official statements from each state 
education agency describing each initiative; 
progress reports on initiative implementation; 
interim or summary studies on each initia- 
tive conducted by states, districts, or outside 
partners; materials describing professional 
development, technical assistance, or other 
support provided by state education agencies; 
and initiative descriptions on service provider 
websites. Information from public documents 
served as the primary data source for the first 



TABLE A2 

State, district, and school background data for the selected sites, 2007-2008 







New York 






Vermont 




Characteristic 


State 


District 

visited 


School 

visited 


State 


District 

visited 


School 

visited 


Total number of districts 


830 


1 


na 


278 


1 


na 


Total number of schools 


4,768 


30 


1 


300 


1 


1 


Total number of teachers 


213,242 


1,600 


40 


8,749 


50 


50 


Total number of students 


2,714,385 


20,000 


300 


90,000 


700 


700 


Student-teacher ratio 


13:1 


11:1 


9:1 


10:1 


12:1 


12:1 


Students eligible for free or 
reduced-price lunch (percent) 


44 


75 


90 


27 


50 


50 


Title 1 eligible 


na 


na 


Yes 


na 


na 


Yes 


Student racial composition (percent) 


Hispanic 


21 


10 


50 


1.1 


2 


2 


Non-Hispanic Asian^ 


7 


2 


0 


1.7 


1 


1 


Non-Hispanic American Indian and Native 
Alaskan 


0.5 


1 


2 


0.3 


0 


0 


Non-Hispanic Black 


19 


50 


30 


1.6 


0 


0 


Non-Hispanic White 


52 


30 


8 


94.0 


90 


90 



na is not applicable. 

Note; To protect confidentiality, data on districts and schools visited are approximate, 
a. Includes non-Hispanic Pacific Islanders. 

Source: Authors' analysis based on data from U.S. Department of Education (2008); New York State Education Department (2009a, b); Vermont Department 
of Education (2008a,b). 
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research question and was used to reduce 
missing interview data for the second research 
question. 

• Interviews. In pairs assigned to each state, 
members of the study team conducted semi- 
structured interviews with two state, two 
district, and two school leaders for each initia- 
tive, using three open-ended interview pro- 
tocols (see appendix E). Interviews were the 
primary data source for the second research 
question and were used to verify data from 
public documents or to address missing data. 

Before being used, each interview protocol was 
pilot tested and revised based on feedback from 
two informants (current or former state, district, 
and school educators familiar with formative 
assessment initiatives outside this project). These 
individuals were identified through professional 
contacts. 



Collection methods 

Data collection, which started in July 2009 and 
was completed in January 2010, included selection 
and review of public documents and interviews 
with two state-level, two district-level, and two 
school-level initiative leaders in each state. 

Terms such as “formative assessment,” “classroom 
assessment,” “student assessment,” and the names 
of each initiative were entered into the search 
engines of state education agency websites and 
online education research clearinghouses and 
databases such as ERIC and Academic Search 
Premier. Study interviewees were also asked for 
any documents containing descriptive informa- 
tion about the initiative and its implementation. 
Documents were examined for data relevant to 
describing the primary components of each initia- 
tive. Data from these documents also provided 
background information that helped in preparing 
for interviews. 

Public documents and professional contacts were 
used to identify the director of the state initiative 



or the initiative’s main contact. Each leader was in- 
vited to participate in an interview, and all agreed. 
Each leader provided the name of a second state 
official who was highly knowledgeable about ini- 
tiative implementation; these officials also agreed 
to participate. Interviews with state officials were 
conducted by pairs of researchers in face-to-face 
formats at state education agency headquarters, 
and each interview lasted 90-120 minutes. With 
the participants’ permission, all interviews were 
digitally recorded and transcribed. 

The state initiative leader in New York was asked 
to recommend at least one official in the district 
implementing the initiative for an interview. The 
recommended district official agreed and was 
asked to recommend a second district official and 
a key school contact. Both agreed to participate. 
The key school contact (the principal) was asked 
to recommend another school leader who was 
knowledgeable about initiative implementation; 
this school leader also agreed to participate. In 
Vermont, the state education agency worked di- 
rectly with schools to implement the initiative. The 
state director was therefore asked to recommend 
key school contacts for interviews. The principal 
of the most highly recommended school agreed 
to participate and recommended a second school 
leader and two district-level leaders — all of whom 
agreed to participate. Interviews with district and 
school leaders were conducted in person by pairs 
of study team members at the district central 
office or school site. Each district- and school- 
level interview lasted 90-120 minutes and was 
digitally recorded and transcribed. A list of the 
jurisdictions and roles of the final interviewees is 
in table A3. 



Data analysis strategy 

Analysis strategies differed for the two research 
questions. 

Research question 1. The study team used a primary 
components analysis matrix for each initiative to 
guide data collection and analysis. Target informa- 
tion from the reviewed documents was inserted 
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TABLE A3 

Jurisdiction and role of state, district, and local 
interviewees, 2009 



Jurisdiction 


Role 


New York 


New York State 


Executive coordinator 


Education Department 


Liaison to district 


District 


Deputy superintendent 




District field coordinator 


School 


Principal 




Math instructional support 
teacher 


Vermont 


Vermont Department of 


Director 


Education 


School support coordinator 


District 


School board member® 




Professional development 
coordinator 


School 


Principal 




Instructional coach 



a. Recommended by the other Vermont district interviewee and the 
school principal, who explained that other district officials were new 
to the district and did not have extensive knowledge of the initiative, 
whereas the district school board member was very knowledge- 
able about the initiative and its implementation history at the school 
selected for this study. 

Source: Authors. 

into the matrix. Data from interviews helped fill 
in missing information and verify information in 
public documents. Questions were sent to inter- 
viewees to clarify additional missing, unclear, or 
contradictory information. Data in the completed 
matrices were compared across the two initiatives 
to identify common and unique patterns. 

Research question 2. All transcribed interviews 
were coded by three researchers using the quali- 
tative data analysis software package ATLAS.ti. 
The coding team began by establishing a list of a 
priori codes, based on the document review and 
interviews. Using these codes, the three study 
team members independently coded sections of 
the same interview. Interrater agreement was 
calculated by dividing the number of times all 
three coders assigned the same code to a para- 
graph of text by the total number of codes or code 



categories assigned by the three coders to that 
paragraph. Percentage agreement for an interview 
section of multiple paragraphs was calculated by 
summing the total number of unanimous code 
use divided by the total instances of individual 
code use over the span of paragraphs within the 
interview section. Initial interrater agreement, 
calculated using an original list of highly detailed 
a priori codes, was very low (15 percent), prompt- 
ing several rounds of discussion among the three 
coders, code clarification and simplification, 
independent recoding of common interview sec- 
tions, and recalculation of interrater agreement 
using a revised list of codes. ^ This iterative process 
continued until the coding team reached an inter- 
rater agreement level of 92 percent for major code 
categories and 70 percent for more detailed codes. 
Appendix F presents the final code list. 

The study team used implementation analysis 
matrices to help analyze coded interview data. The 
matrices arrayed coded interview data by major 
categories and by the two initiatives. Data in the 
matrices were compared across the two initiatives 
to identify similarities and differences in initia- 
tive structure and approach. Common patterns 
were identified and agreed on through discussion 
and verification of interview transcript and public 
document data. 



Confidentiality 

The study team worked closely with the Institu- 
tional Review Board at Education Development 
Center, Inc. to ensure that interviewees knew their 
rights as study participants. The team focused 
on collecting data on policies and practices at 
the state, district, and school levels and did not 
request personal information. Interviewees 
were informed that the study would protect the 
confidentiality of their responses and signed an 
informed consent form. The information inter- 
viewees provided was kept secure and shared only 
among the study team. Research data were not 
labeled with interviewees’ real names, and their 
names were omitted or replaced by pseudonyms in 
interview transcripts and internal reports. 
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APPENDIX B 

TEACHER PRACTICES PROMOTED 
BY EACH INITIATIVE 

Both the New York State Formative Assessment 
Technical Assistance Study (NY-FATA) and the 
Vermont Formative Assessment Project (VT-FAP) 
promoted formative assessment practices consis- 
tent with the Council of Chief State School Officers 



(CCSSO) definition (table Bl). NY-FATA adopted the 
CCSSO definition but used descriptions of practices 
from multiple sources in written training materials. 
VT-FAP adopted both the definition of formative 
assessment and descriptions of its practices from 
Keeping Learning on Track (KLT) program materi- 
als, which are consistent with the CCSSO definition. 
Both sources emphasize the teacher’s role and an 
active student role in student learning. 



TABLE Bl 

Formative assessment definition and practices in the New York State Formative Assessment Technical 
Assistance Study and the Vermont Formative Assessment Project, 2007-10 



Definition and 
characteristics 



Definition 



Practices 



New York State Formative Assessment 
Technical Assistance Study 



"A process used by teachers and students during 
instruction that provides feedback to adjust 
teaching and learning to improve students' 
achievement of intended instructional outcomes" 
(McManus 2008, p. 3). 

Five attributes 

• Learning progressions clearly articulate the 
subgoals of the ultimate learning goal. 

• Learning goals and criteria for success are clearly 
identified and communicated to students. 

• Students are provided with evidence-based 
feedback linked to the intended instructional 
outcomes and criteria for success. 

• Both self- and peer-assessment are Important 
for providing students an opportunity to think 
metacognitively about their learning. 

• The classroom culture treats teachers and 
students as partners in learning (McManus 2008). 



Vermont Formative Assessment Project 



"Students and teachers using evidence of learning 

to adapt teaching and learning to meet immediate 

learning needs minute-to-minute and day-by-day" 

(Educational Testing Service 2007, p. 25). 

Five strategies (Leahy et al. 2005, p. 20) 

• Clarifying and sharing learning intentions and 
criteria for success. 

• Engineering effective classroom discussions, 
questions, and learning tasks. 

• Providing feedback that moves learners forward. 

• Activating students as the owners of their own 
learning. 

• Activating students as instructional resources for 
one another. 



Source: Authors' analysis based on McManus (2008) and Educational Testing Service (2007). 
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APPENDIX C 

PRIMARY COMPONENTS OF EACH INITIATIVE 



TABLE Cl 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



Component 



Duration 



Goals® 



Origins 



New York State Formative Assessment Vermont Formative 

Technical Assistance Study Assessment Program 



A three-year pilot project, from 2007 to 2010. An ongoing initiative launched In 2006/07. Its first 

year was considered Its pilot year. 



Improve student achievement through teacher 
use of formative assessment in the classroom. 

Establish whether the district could use formative 
assessment to closely monitor student progress 
and help teachers analyze and respond to 
individual student learning needs. 

Improve teacher skills in math through 
professional development in math content and 
pedagogy and the use of formative assessment in 
math instruction. 

Support formative assessment practices in subject 
areas other than math. 

Build district Internal capacity to support teacher 
use of formative assessment In pilot schools and 
throughout the district. 

Build a shared understanding of formative 
assessment across different departments of the 
New York State Education Department. 

Help the New York State Education Department 
learn whether the formative assessment Initiative 
could be an effective school Improvement 
strategy, and If so, build departmental capacity to 
support teacher use of formative assessment in 
the pilot district and throughout the state. 



Improve student achievement by supporting the 
integration of formative assessment into teacher 
instructional practice. 

Support school improvement by establishing 
teacher learning communities (TLCs), an 
organizational structure to help teachers build 
collective responsibility for student learning 
and, in the context of VT-FAP, to help them build 
knowledge of and experience with formative 
assessment. 

Support and promote use of formative assessment 
in schools throughout the state. 

Build state capacity to provide basic training in 
formative assessment to schools throughout the 
state, without need for support from external 
partners. 

Build state capacity to support a train-the-tralner 
system to sustain and spread VT-FAP In the event 
of turnover or attrition within the existing state- 
based initiative training corps. 



The New York State Education Department 
encouraged an urban district to apply for 
Mathematics and Science Partnership grant funding 
to launch a formative assessment Initiative, which 
would receive technical support from the New York 
Comprehensive Center (NYCC) and the Accountability 
and Assessment Comprehensive Center (AACC). The 
urban district pursued support for this initiative in 
response to several perceived needs: 

• Better teacher math instruction. 

• Closer monitoring of student progress to 
understand and respond to individual student 
learning needs. 

• Higher student achievement. 

The state was interested in learning whether 
the initiative could serve as an effective school 
improvement model for districts in the state. 



The pilot was launched by the Vermont Department 
of Education in response to two perceived needs at 
the local level: 

• Better guidance for designing local assessment 
plans with a strong foundation In classroom 
formative assessment. 

• Higher student achievement. 

Department staff, based on their reading of the 
research literature, believed that building teacher 
capacity in formative assessment could provide a 
strong foundation for improving student learning. 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment 
Component Technical Assistance Study 



Leaders and roles 

State New York State Education Department, Department 

education of Curriculum, Instruction, and Instructional 

agency Technology: 

• Brokered the relationship between district leaders 
and external partners. 

• Identified initiative funding. 

• Observed implementation. 

• Received NYCC and AACC technical assistance 
to build shared understanding of formative 
assessment within the state education agency 
and to consider policies and practices that would 
support the spread of formative assessment, 

as defined by the Council of Chief State School 
Officers (CCSSO), to other districts. 



District • Conceptualized and designed the initiative with 

NYCC staff. 

• Provided oversight of the initiative at the district 
level. 

• Worked to tie the initiative into ongoing district 
supports for teachers. 

External New York Comprehensive Center (NYCC) 

partners , Flelped the district design the initiative (with 

support from the Assessment and Accountability 
Comprehensive Center). 

• Developed math coach training with district Input. 

• Provided technical assistance to state officials and 
district leaders. 

Assessment and Accountability Comprehensive Center 

(AACC) 

• Provided access to a national expert to help the 
NYCC develop its role in the initiative. 

• Flelped the NYCC facilitate two summer Institutes 
and occasional training sessions during the school 
year. 



Vermont Formative 
Assessment Program 



Vermont Department of Education Standards and 

Assessment Team: 

• Designed the initiative. 

• Provided oversight. 

• Selected an external partner to provide 
professional development for school-based 
groups of teachers, school administrators, and 
state staff. 

• Flired eight coaches to serve as TLC facilitators for 
the pilot year. 

• Provided summer institute training through state 
staff and the eight coaches for new groups of TLCs, 
principals, and other local administrators starting 
the second year. 

• Collaborated with educational services agencies — 
partnerships among school districts, unions, 
institutions of higher education, and service 
providers to offer professional development 

to meet the regional needs of educators — to 
organize logistics of summer institute trainings. 



Educational Testing Service (ETS) 

• Designed the professional development program 
Keeping Learning on Track (KLT), a two-year, 
16-module training curriculum in formative 
assessment for TLCs. 

• Provided an introductory summer training 
institute during the pilot year for state staff, the 
eight coaches who would serve as TLC facilitators, 
and participating teacher groups. 

• Trained state staff and the eight coaches during 
spring of the pilot year to serve as summer 
institute trainers. 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment Vermont Formative 

Component Technical Assistance Study Assessment Program 



• State invited Title I schools to participate in the 
pilot. Several non-Title I schools also applied and 
were accepted because the project budget was 
able to accommodate them. 

• Nine schools participated in the pilot year. 

• Since year 3, participation has been encouraged or 
required for: 

• Schools in restructuring under the No Child 
Left Behind Act (11 schools in 2008) are 
required to participate in VT-FAP or adopt 
another school intervention program (Critical 
Friends Groups). 

• Schools in districts that apply for the state- 
administered federal Title ll-B Mathematics 
and Science Partnership grant are required 
to include VT-FAP participation in their 
applications. 

• By spring of year 4, VT-FAP was active in 55 
schools. 

Targeted • Year 1: Grades 4 and 5 (one participating school Open to teachers of any grade level (K-12). 

grades instead targeted kindergarten and grade 1). 

• Years 2 and 3: Grades 4 and 5 no longer the 
exclusive focus; grade levels varied by school. 

Targeted Math 

content area 

Funding 

• Title ll-B Mathematics and Science Partnership. 

• NY-FATA was one component of a larger initiative 
supporting math content knowledge and 
pedagogy. 



All content areas. 



Year 1 : Title I, Title ll-A, and GEAR UP. 

Year 2 and after, local funds cover summer 
institute costs, including materials, space, and 
trainer if no state trainer is available. 



Targeted participants 

Targeted • District leaders invited elementary school 

schools principals in the pilot district to participate; 10 

principals volunteered their schools. 

• Nine schools participated; one school did not 
participate because of teacher resistance. 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment 
Component Technical Assistance Study 



Professional development 

Teachers Summer institute providers: NYCC and AACC, but 

teachers did not participate in the year 1 and 2 

summer Institutes. 

Ongoing training/support providers: Math coaches 

trained by the NYCC. 

Training/support content: 

• Year 1: Content delivered in monthly group 
meetings covered background, theory, and 
classroom strategies of formative assessment. 
Content delivered by math coaches mirrored 
the content they had learned from the NYCC the 
previous month. 

• Years 2 and 3: Content varied by teacher need. 

Nature and structure of training/support: 

• Year 1 : Mandatory participation by grade 4 and 5 
teachers (in eight schools) and kindergarten and 
grade 1 teachers (in one school) in monthly two- 
hour group meetings with the school math coach 

• Years 2 and 3: Teachers received training through 
one of two formats: 



Vermont Formative 
Assessment Program 



Summer institute providers: 

• Year LETS. 

• Year 2 and later: State staff or one of the eight 
original coaches trained as summer institute 
trainers. Institutes have been organized and 
administered through regional educational 
services agencies. 

Ongoing training/support providers: TLCs. 

• Year TTLCs in participating schools were 
facilitated by one of the eight coaches hired by the 
state. 

• Year 2 and later: TLCs facilitated by school-based 
teachers or coaches trained the previous summer 
to serve as TLC facilitators. 

Training/support content: 

• Summer institute introduces participants to 
the background, theory, and research behind 
formative assessment and strategies for its 
practice in the classroom, and to TLCs and the KLT 
curriculum. 



Voluntary one-on-one or small-group work 
with school math coach, as needed or as part of 
an action research project for teachers' yearly 
evaluation. 

Voluntary enrollment in a 30-hour after- 
school course developed by district math 
coaches. Formative Assessment 1 covered 
basic concepts of formative assessment; 
Formative Assessment 2 covered formative 
assessment classroom strategies. Courses were 
administered through the district's Teacher 
Center. 



• Content of monthly TLCs follows the KLT program 
curriculum: 16 modules over two years. 

Nature and structure of training/support: 

• Mandatory participation in an introductory 
summer training session (two days). 

• Mandatory participation in monthly TLCs for two 
years. After two years, TLCs can continue to meet 
and follow their own curriculum at their discretion 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment 
Component Technical Assistance Study 



Professional development (continued) 

Teacher Audience: Math coaches. 

leaders Summer institute providers: NYCC and AACC. 

Ongoing training/support providers: NYCC (with 

occasional support from the AACC); math coach 

peers. 

Training/support content: 

• Summer institute introduces participants to 
the background, theory, and research behind 
formative assessment and to strategies for its 
practice in the classroom. 

• Year 1: Ongoing training in formative assessment 
concepts and strategies from the NYCC and on 
how to transfer this knowledge to teachers. 

• Year 2: Feedback from NYCC consultants and math 
coach peers after observations of Individual math 
coaches' work with teachers in classrooms. 

• Year 3: Content varied by individual math coach 
need. 

Nature and structure of training/support: 

• Mandatory participation in summer training 
institutes in year 1 (one day) and year 2 (two days). 

• Year 1 : Mandatory monthly meetings with NYCC to 
gain basic training in formative assessment and in 
how to transfer this training to teachers. 

• Year 2: Mandatory monthly or bimonthly meetings 
with NYCC and math coach peers, to observe and 
receive feedback on coaching efforts with teachers 
in classrooms. 

• Year 3: Voluntary informal support available from 
the NYCC by phone or email as needed. 



Vermont Formative 
Assessment Program 



Audience: JLC facilitators. 

Summer institute providers: 

• YearTETS 

• Year 2 and later: A state staff member or one of 
the eight original coaches trained as KLT trainers. 
Institutes have been organized and administered 
through regional educational service agencies. 

Ongoing training/support providers: 

• Year 1 : The eight coach peers, with oversight from 
the state. 

• Year 2 and later: TEC facilitator peers. 

Training/support content: 

• Summer Institute training introduces participants 
to the background, theory, and research behind 
formative assessment and to strategies for its 
practice in the classroom. TLC facilitators also 
receive training in how to lead TLCs through the 
KLT curriculum over the next two years. 

• Year 1 : Ongoing reflections and discussions about 
TLC leadership and issues encountered during 
initiative implementation. 

• Year 2 and later: Content varies by individual TLC 
facilitator need. 

Nature and structure of training/support: 

• Mandatory participation in a two-day summer 
training institute and an additional day of training 
for TLC facilitators (two days during the pilot year). 

• Year l:The eight coaches hired as TLC facilitators 
were required to participate in a state-moderated 
online community and to meet face-to-face 
several times during the year. 

• Year 2 and beyond: School-based TLC facilitators 
can meet voluntarily and collaborate with peers 
through an state-moderated online community 
Some educational services agencies have held 
mid-year meetings for TLC facilitators to provide 
opportunities for peer support. 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment Vermont Formative 

Component Technical Assistance Study Assessment Program 



Professional development (continued) 



Principals 



Summer institute providers: NYCC and AACC. 

Ongoing training/support providers: District officials 

with NYCC. 

Training/support content: 

• Content of summer institute training same as for 
math coaches. 

• Support provided during periodic meetings held 
by district leaders and NYCC includes updates 
on Initiative Implementation. Other topics vary 
according to the issues principals raise. 

Nature and structure of training/supports: 

• Optional participation in year 1 and 2 summer 
training institutes and initiative information 
sessions. 

• Optional periodic meetings of principals or 
designees with district and NYCC, to stay informed 
about the initiative and discuss issues of concern 
to principals. 



Summer institute providers: 

• YearTETS. 

• Year 2 and later: A state staff member or one of 
the eight original coaches trained as KLT trainers. 
Institutes have been organized and administered 
through regional educational services agencies. 

Ongoing training/support providers: State leaders and 

fellow principals participating in mandatory state 

meetings. 

Training/support content: 

• Content of summer institute training the same as 
for teacher groups and TLC facilitators. 

• Support during regular meetings held by state 
varies according to the issues principals raise. 

Nature and structure of training/supports: 

• Mandatory Introductory summer training (two 
days). 

• Mandatory meetings with state education agency 
for principals or designees of nine original pilot 
schools (year 1) and for schools in restructuring 
(year 3 and later) for support and communication 
about the initiative. 



District 

leaders 



Summer institute providers: NYCC and AACC. 

Ongoing training/support providers: NYCC. 

Training/support content: 

• Content of summer Institute training same as for 
math coaches and principals. 

• Regular meetings with NYCC at end of year 1 
focused on designing year 2 training and support 
for math coaches as well as ongoing development 
of the overall initiative. 

Nature and structure of supports: 

• Years 1 and 2: District leaders voluntarily 
participated In summer institutes and monthly 
training sessions held by the NYCC for math 
coaches. 

• Periodic meetings with NYCC staff to develop the 
initiative. 



No special professional development, training, or 
support for district leaders. Summer institute training 
is available to all educators in the state wishing to 
participate. 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Primary components of New York and Vermont formative assessment initiatives, 2007-10 



New York State Formative Assessment 
Component Technical Assistance Study 



Professional development (continued) 

State leaders Summer institute providers: NYCC and AACC. 

Ongoing training/support providers: NYCC. 

Training/support content: 

• Content of summer Institute training same as for 
math coaches, principals, and district leaders. 

• Regular meetings with NYCC cover the theory 
and research behind formative assessment. 
Discussions also involve strategies for scaling the 
initiative to other districts. 

Nature and structure of supports: 

• Years 1 and 2: State leaders were Invited to 
participate in summer training institutes. A 
working group of state leaders from different 
departments in the New York State Education 
Department also attended regular meetings with 
the NYCC. 



Formal evaluations 

• "Title ll-B Mathematics and Science Partnership: 
Year One Report, 2007-08" (Evaluation 
Consortium, University at Albany-SUNY, 2008). 

• "New York Formative Assessment Project: From 
Research to Practice" (New York Comprehensive 
Center, 2009). 

• Upcoming evaluation for the district board 
curriculum committee (no date available) by an 
external evaluator. 



Vermont Formative 
Assessment Program 



Summer institute providers: ETS (year 1 only). 

Spring training session providers: ETS (year 1 only). 

Ongoing training/support providers: None. State 
officials and coaches trained as summer Institute 
trainers try to keep their training skills fresh by 
leading a summer training institute with other in- 
state trainers each year. 

Training/support content: 

• Content of summer institute training in year 1 the 
same as for the eight coaches hired by the state as 
TLC facilitators. 

• Year 1 : Spring training session trained state leaders 
and the eight coaches as summer institute trainers. 
These staff learned to deliver the same training 
that ETS had provided to them the previous 
summer. State staff and the eight coaches were 
also trained to train teachers and school-based 
coaches as TLC facilitators. 

Nature and structure of supports: 

• Year 1: Summer institute training lasted two days 
for all participants and an additional two days for 
state staff (approximately 25 attended) and the 
eight coaches. 



External consultants conducted a qualitative 
evaluation of the pilot year implementation, to 
inform continuation of the project. Findings have 
been published by the Vermont Department of 
Education on its website (Cole and O'Brian 2007). 

For participating schools undergoing 
restructuring, the Vermont Department of 
Education has commissioned a survey from 
an outside university to examine teachers' 
experiences with implementation of the initiative 
and levels of collaboration. 



a. Perspectives revealed through public sources, interviews, and documents provided by state, district, and school interviewees. Goals listed do not include 
the perspectives of external initiative partners, which were not examined in this study. 

Source: Compiled by authors from public sources, interviews, and documents provided by interviewees, as described in the text. 
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APPENDIX D 
RESEARCH CONTEXT ON 
FORMATIVE ASSESSMENT 

This overview briefly describes the literature on 
formative assessment, including research on the 
relationship between formative assessment and 
student learning, difficulties promoting its use 
among teachers, and strategies to promote sus- 
tained and widespread use. 



Defining formative assessment 

Formative assessment has become a “buzzword 
in education” (McMillan 2007, p. 1) as the use of 
large-scale assessments has increased and forma- 
tive assessment has been proposed as a means of 
raising student achievement (Black and Wiliam 
2003; Chappuis and Chappuis 2007/2008; McManus 
2008; Tanney et al. n.d.). Flowever, educators have 
used vague and inconsistent definitions of formative 
assessment (Black and Wiliam 1998a; Chappuis and 
Chappuis 2007/2008). The CCSSO convened state 
education agency leaders and experts in formative 
assessment to bring greater clarity to the topic. They 
arrived at the following consensus definition: 

Formative assessment is a process used by 
teachers and students during instruction 
that provides feedback to adjust ongoing 
teaching and learning to improve students’ 
achievement of intended instructional out- 
comes (McManus 2008, p. 3). 

This definition, applicable to a wide spectrum of 
practices, emphasizes formative assessment as an 
ongoing process that is embedded in instruction 
and that provides evidence to inform changes 
in instruction (McManus 2008). Central to this 
definition is the notion that “learning requires 
active and frequent interaction between teach- 
ers and learners” (Black et al. 2004, p. 47). The 
definition also points to an active role for students. 
Through classroom discussions and self- and peer- 
assessment activities, formative assessment can 
help students feel more responsible for their own 
learning (McManus 2008). 



Formative assessment differs from other types 
of assessment in being an ongoing process 
rather than a singular event (McManus 2008), 
requiring active exchange between teachers and 
students (Black et al. 2004), and leading teach- 
ers to adapt instruction to student needs (Black 
and Wiliam 1998a). Thus, how teachers use 
assessment data is critical. Black and Wiliam 
(1998a) argue that any assessment may provide 
information to guide classroom instruction but 
that assessment is formative only “when the 
evidence is actually used to adapt the teach- 
ing to meet student needs” (para. 7). Formative 
assessment activities must provide data that 
identify a student’s status in relation to a specific 
learning goal and the steps the student can take 
to reach it (Heritage 2007; Wiliam and Leahy 
2007). In contrast, summative assessments are 
generally administered at the end of a specific 
unit of time to “evaluate a student’s performance 
against a defined set of content standards” (Perie 
et al. 2007, pp. 1-2). Some researchers argue 
that other practices sometimes classified as 
formative are more accurately labeled interim, 
frequent, benchmark, predictive, or early warn- 
ing summative assessment (Black and Wiliam 
2005; Chappuis 2005; Perie et al. 2007). They 
may provide feedback for adjusting instruction 
but do not involve classroom-based interactive 
instructional assessment strategies. 



Evidence of the effects of formative assessment 

Evidence suggests that classroom-based forma- 
tive assessment can promote student learning if 
properly implemented (Black and Wiliam 1998a; 
Furtak et al. 2008). Some studies have associ- 
ated formative assessment with the development 
of metacognitive skills as well as greater learn- 
ing in math and science (Assessment Reform 
Group 2002; Shepard et al. 2005; Wiliam 2007). 

A randomized controlled trial study of “running 
records,” a formative assessment technique in lit- 
eracy instruction, showed statistically significant 
increases in student achievement compared with 
use of a nonformative assessment intervention 
(Ross 2004). 
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Other research suggests that formative assess- 
ment may produce greater increases in student 
achievement — at a lower cost — than smaller class 
sizes or greater teacher content knowledge (Wil- 
iam and Thompson 2007). A study comparing the 
cost- effectiveness of different intervention models 
suggests that for every dollar spent on interven- 
tions, student outcomes in reading and math are 
greater for rapid assessment than for comprehen- 
sive school reform and high-quality preschool (Yeh 
2008). 

Despite the positive findings, researchers (Black 
and Wiliam 1998a; Furtak et al. 2008; Shepard 
2005; Yin et al. 2008) caution that simply embed- 
ding formative assessment into the curriculum 
is not enough; teachers need support and profes- 
sional development, and success depends on the 
context in which the practice is implemented. They 
also caution that much of the evidence comes from 
laboratory and anecdotal studies and that studies 
with an experimental design in regular classroom 
settings are lacking (Black and Wiliam 1998a; Yin 
et al. 2008). 



Implementation challenges 

Engaging in formative assessment may require 
a shift in attitudes about teaching and learning, 
away from traditional views of teachers as trans- 
mitters and students as recipients of knowledge. 
Students participating in classroom formative 
assessment learn through inquiry and exchange 
in partnership with teachers and other students 
(Black et al. 2004; Black and Wiliam 1998b; 
Heritage 2007). Many teachers must also learn 
to construct and interpret assessments, to teach 
students to assess their own work and set their 
own learning goals, to shift from correcting to 
informing students, and to use effective question- 
ing to engage student thinking (Black and Wiliam 
1998a; Brookhart 2007; Chappuis and Chappuis 
2007/2008; Heritage 2007; Moss and Brookhart 
2009). Teachers must also learn to use information 
collected from students to identify instructional 
approaches that meet student needs (Heritage 
et al. 2009). 



Changing teachers’ basic conceptions of the roles 
of teachers and students and their basic meth- 
ods of instruction is difficult (Elmore 2004), as 
is acquiring specific formative assessment skills 
(Heritage et al. 2009; Moss and Brookhart 2009). 
Furthermore, education leaders trying to promote 
formative assessment through organized pro- 
grams face common implementation challenges 
in program adoption, sustainability, and spread 
(see box Dl). Promoting formative assessment 
therefore requires not only changing fundamen- 
tal teaching and learning practices in individual 
classrooms, but also building organizational 
structures to help execute and scale up formative 
assessment practices. 



Strategies to promote and spread fundamental 
changes in teaching and learning 

Because formative assessment often requires 
fundamental shifts in teaching and learning, 
and because research suggests that engineer- 
ing such shifts in lasting ways has been elusive 
(Elmore 2004), promoting formative assessment 
may require initiatives that address scale and 
sustainability from the start. Strategies that may 
facilitate program adoption, sustainability, and 
scaling among teachers and school administrators 
include: 

• Selecting sites carefully. The likelihood of 
engaging teachers and school leaders in 
education reform initiatives rises when reform 
leaders and designers choose implementation 
sites that take reforms seriously and have the 
capacity to support reforms after involve- 
ment by external partners ends (Eixsen et al. 
2005; McLaughlin and Mitra 2001; Supovitz 
2008). Strong implementation sites are gener- 
ally those where teachers and administra- 
tors agree on the fundamental problems in 
teaching and learning and see the proposed 
reforms as a good match for solving them 
(Supovitz 2008). 

• Aligning initiatives and school or district 
policy. Schools and districts are often buffeted 
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BOXD1 

Implementing new programs 

A review of research in several fields 
identifies multiple stages to program 
implementation: adoption, initial 
implementation, full operation, in- 
novation, and sustained implementa- 
tion (Fixsen et al. 2005). Program 
adoption and initial implementation 
can be impeded by “fear of change, 
inertia, and investment in the status 
quo combine [d] with the inherently 
difficult and complex work of imple- 
menting something new” (p. 16). 
Reaching full operation requires help- 
ing staff and organizations integrate 
new learning into daily practices and 
routines. During the innovation stage, 
program leaders must learn to recog- 
nize and minimize deviations from 



core practices that could undermine 
desired outcomes, while allowing in- 
novations that facilitate full program 
operation. During the final phase of 
implementation, program leaders 
must tackle threats to program sus- 
tainability, from turnover of staff to 
changes in organizational leadership, 
program funding, and community 
priorities (Fixsen et al. 2005). 

Achieving sustainability and scal- 
ing up programs are often related. 
Researchers frequently refer to scaling 
up as extending a program to popula- 
tions and sites beyond initial pilot 
groups, to benefit as many people as 
possible (Fixsen 2009; Fixsen et al. 
2009). Some researchers suggest that 
program scaling requires attention to 
issues such as depth, sustainability. 



and ownership of program implemen- 
tation as well as total numbers of tar- 
get groups adopting the program (Co- 
burn 2003; Fixsen 2009; McLaughlin 
and Mitra 2001). Education leaders 
who wish to extend education reforms 
both broadly and deeply therefore face 
multiple challenges, such as getting 
larger organizational systems (like 
schools, districts, and states) to in- 
corporate reform principles into their 
regular norms of practice, providing 
individuals at all levels of the sys- 
tem with opportunities to learn and 
develop new reform skills, procuring 
adequate resources to support pro- 
gram expansion, and building capac- 
ity within local sites to lead, manage, 
and assume responsibility for ongoing 
program implementation (Coburn 
2003; McLaughlin and Mitra 2001). 



by competing initiatives and priorities, which 
can dissuade teachers and administrators 
from adopting and supporting reforms. While 
reform leaders may wish to choose receptive 
implementation sites, they may also wish 
to work with school and district officials to 
identify how proposed reforms and existing 
policies and initiatives can complement each 
other (McLaughlin and Mitra 2001). Aligning 
reform practices and existing priorities may 
help convince school and district leaders to 
support the reform (Coburn 2003). 

• Working with school and district communica- 
tion channels. Existing networks and organi- 
zational structures (such as grade-level teams, 
professional learning communities, and webs 
of informal relationships) may be important 
sources of professional information, influence, 
and ongoing learning for teachers. To assess 
whether support for reforms will be most 
effective through new or existing community 
structures, reform leaders may need to under- 
stand how ideas and practices spread within 



particular settings. The channels through 
which reforms enter an organization could 
affect how reforms are adopted, adapted, and 
sustained (Supovitz 2008). 

• Providing ongoing professional development 
and support. Achieving deep, sustained, and 
pervasive teacher change may require ongoing 
professional development, new teacher support 
structures, and expectations for practice that 
extend across levels of the education system 
(Elmore 2004). Detailed guidance materials, 

a community of supportive peers, and knowl- 
edgeable leaders who are supportive of teach- 
ers’ change efforts may be especially important 
when reforms are complex and changes to 
teacher practice are fundamental (Coburn 
2003; Supovitz 2008). Ongoing teacher training 
may be most effective when trainers are deeply 
knowledgeable about the initiative’s principles 
and practices (Supovitz 2008). 

• Developing local leaders to support and spread 
the initiative. District and school leaders play 
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an important role in reform implementation 
by aiding reform efforts and communicating 
the importance of adopting and sustaining 
change (Supovitz 2008). If reform leaders 
are from outside the districts and schools, 
research suggests that these external partners 
can help to build capacity among local leaders 
to carry on the reform independently over 
the long term (Coburn 2003; McLaughlin 
and Mitra 2001). Building such capacity may 
include clearly delineating initiative roles and 
responsibilities for school and district admin- 
istrators from the beginning, modeling how 
to incorporate reform principles and practices 
into local leaders’ own work and broader 
education policies, and developing a group of 
knowledgeable teacher leaders who can train 
and provide ongoing support to teachers in 
using reform practices (Coburn 2003; Supo- 
vitz 2008). 

• Securing ongoing funding. Once initial 

program funding ends, ongoing funding is 
needed to train and support teachers and 
administrators who are working to change 
teaching and learning practices. Additional 
funding may also be needed to spread reform 
practices beyond initial sites. Reform leaders 
may want to build local capacity to develop 
future streams of funding to support and 
expand reform practices (Coburn 2003). 



How state education agencies can 
promote formative assessment 

Since formative assessment is embedded in daily 
instruction (Wiliam and Leahy 2007), it is reason- 
able to ask what state education agencies can do 
to promote formative assessment practices at the 
school and classroom level (Black and Wiliam 



2003; Darling-Hammond and McLaughlin 1995; 
Stiggins 2002; Tanney et al. n.d.). To meet the 
No Child Left Behind Act goal of proficient per- 
formance on state assessments for all students by 
2014, state education agencies need to find ways 
to improve the performance of low-performing 
districts and schools, a departure from their tra- 
ditional roles of disbursing grants and monitoring 
compliance with federal requirements (Lusi 1997). 

State education agencies may need to build their 
own capacities to support effective instruction and 
professional development and to adopt new ways 
of communicating and working with districts and 
schools to assist with school improvement efforts 
(Sunderman and Orfield 2006). To initiate, design, 
and lead formative assessment or other classroom- 
level reform efforts, states may need to acquire 
the resources to execute new functions. They 
can identify and secure the assistance of strong 
external partners to provide schools and districts 
with the expertise and training to implement 
reforms (Lusi 1997), connect schools and districts 
to help them share and learn from one another’s 
experiences (Perie et al. 2007), and provide tradi- 
tional supports. Supports may include allocating 
federal funds and requesting legislative funding 
to reinforce change (Perie et al. 2007); monitoring, 
evaluating, and conducting research on reform 
implementation (Black and Wiliam 1998b); and 
establishing state-defined learning standards 
or grade-level expectations to guide assessment 
activities (Perie et al. 2007). Many states also have 
the opportunity to influence policies that affect 
implementation of formative assessment. For ex- 
ample, as authorizers of higher education teacher 
training programs, state education agencies may 
be able to ensure the integration of formative as- 
sessment into preservice programs (Rindone and 
MacQuarrie 2010). 
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APPENDIX E 
INTERVIEW PROTOCOLS 

This appendix contains the protocols used for 
interviews with state, district, and school initiative 
leaders in New York and Vermont. 



State initiative leaders 

Introductory script 

Note: recorder should be on if consent granted 

Thank you for agreeing to speak with us today 
about your state’s efforts to promote formative 
assessment. 

We are . . . [introduce selves]. 

We understand that your state has launched an 

effort called the project. Our goals for 

this interview are to learn about: 

a. the goals of the proj ect; 

b. the types of efforts the state has made to imple- 
ment or support the project; and 

c. how effective you think state and local efforts 

have been to implement the 

project. 

Do you have any questions before we begin? 

I. Individual background 
First, some questions about you. 

1 . What is your role in the state education agency 
(SEA)? 

2. How did you personally get involved with the 
project? 

3. How does your work for the proj- 

ect relate to your other state responsibilities? 

II Project goals 

Now, we would like to learn more about the 
project. 



4. What is the working definition of “formative as- 
sessment” that the state is using for your project? 

Probes: 

Under this definition, what roles do teachers 
play in formative assessment? 

Students? Parents? 

5. How did the state arrive at its working defini- 
tion of formative assessment? 

6. What would you say are the main goals of the 
project? 

Probe: 

What were the original motivations for 
launching the project? 

7. What evidence does or will the state use to 
determine whether the project is a success? 

8. How is this project related to other state 
assessment, instruction, or other school im- 
provement efforts? 

III. Methods and opinions of project 
implementation 

9. Restate the project goals they identified earlier. 
To what extent do you think these goals have 
been achieved? 

10. In the following series of questions. I’m going 
to ask you to describe how the state has imple- 
mented the project, and how well you think 
different implementation efforts have worked. 

What efforts has the state used: 

a. To reach a shared understanding of for- 
mative assessment practices? 

i. How well do you think these imple- 
mentation efforts have worked? 

Discuss each effort separately. 

At end, ask: Which methods worked 
very well, moderately well, or not so 
well? 
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ii. In your view, what factors have led 
these efforts [list which ones] to work 
very well? 

Repeat for efforts listed under moder- 
ately well and not so well. 

b. To promote the types of assessment prac- 
tices the state would like to see within 
classrooms? [within target content areas, 
grade levels; at desired scale] 

Probe, if not discussed: What has the state 
done to promote formative assessment, 
as defined by the CCSSO? (which involves 
more constant assessment and adjust- 
ments in instruction) 

c. To establish and maintain the project’s 
schedule or timeline? 

d. To fund the project? 

e. To provide professional development and/ 
or technical assistance for the project? 

f To monitor and/or evaluate the project? 

11. We’d like to ask a few more questions about 
support provided for the project, focusing 
on professional development and technical 
assistance, whether facilitated by the state or 
external partners. 

a. What has been the content of professional 
development for the project? 

Provide examples if needed, e.g., how to 
ask good questions, how to provide feed- 
back, how to find time to conduct forma- 
tive assessment. 

b. Who have been the target audiences for 
professional development? 



c. What has been the structure of profes- 
sional development? 

Probe both for when, e.g, during or 
outside school daylyear, and format, e.g, 
workshops; in-class coaching. 

12. We also have a couple questions about techni- 
cal assistance for the project. 

a. What types of technical assistance has 
the state (and external partners, if rel- 
evant) provided for the project? 

Provide examples if needed, e.g, data 
management, data analysis. 

b. Who have been the target audiences for 
technical assistance? 

IV. Project participants 

We would like to learn who is involved in this 
project and how they became involved. 

13. How did your department or office in the SEA 
get involved with the project? 

Probe if not discussed: Is the state involved in 
supporting formative assessment practices 
outside this project? 

If yes: Can you describe these efforts? 

How did the SEA get involved in 
them? 

14. What other offices in the SEA have collabo- 
rated on this project? 

15. What other organizations, outside the SEA, 
have collaborated in this project? 

16. How did districts or schools come to partici- 
pate in this project? 

Probe if not discussed: Is participation related 
to a district or school’s No Child Left Behind 
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status (i.e., whether they are in corrective ac- 
tion)? If yes, please explain. 

17. How engaged have participating districts and 
schools been in this work around formative 
assessment? 

Follow up with: 

• Please describe the ways in which par- 
ticipating districts and schools have been 
engaged. 

• In your mind, what factors have con- 
tributed to greater district and school 
engagement? 

V. Participant roles 

We are interested in learning more about the guid- 
ance and support the state has provided for this 
project, and the roles played by other participants. 

If SEA has worked with external partners, ask Q18. 
Otherwise, skip to Q19. 

18. How has the SEA worked with external 
partner organizations to support formative 
assessment practices? 

Probe if not discussed: How did the SEA 
choose these partners? 

19. What kind of guidance or support has the 
state provided to districts over the course of 
this project? 

If not discussed, probe for guidance/support 
given by his/her SEA office and other SEA offices. 

20. What kind of guidance or support has the 
state provided to schools over the course of 
this project? 

If not discussed, probe for guidance/support 
given by his/her SEA office and by other SEA 
offices. 



21. What kind of guidance or support have dis- 
tricts provided to schools over the course of 
this project? 

If state has worked with external partners, ask 
Q22 and Q23, otherwise skip to Q24. 

22. What kind of guidance or support have ex- 
ternal partners provided to districts over the 
course of the project? 

23. What kind of guidance or support have 
external partners provided to schools over the 
course of the project? 

24. What efforts has the state used to establish 
and maintain partner roles and relationships 
(state staff, partners, etc.) 

Probes 

• How well do you think these implementa- 
tion efforts have worked? Discuss each 
effort separately. Ask which ones worked 
very, moderately, or not so well. 

• In your view, what factors have led these 
efforts [list which ones] to work very well? 
Repeat for efforts listed under moderately 
and not so well. 

VI. Future directions 

Now we have questions about the future of the 
project and formative assessment efforts in your 
state. 

25. What are the state’s plans for formative as- 
sessment practices in the next year? The next 
3-5 years? 

Probe, if not discussed: How does the state 
plan to expand the use of formative assess- 
ment over time? 
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26. What would you like to see happen with 
formative assessment in your state in the next 
year? The next 3-5 years? 

27. Knowing what you know now, what advice 
would you give other SEAs that want to 
promote formative assessment practices (as 
defined by the CCSSO) at a large scale [build- 
ing teacher capacity statewide]? 

28. What do you think are the best roles that 
states can play to support the use of formative 
assessment (as defined by the CCSSO) in class- 
room instruction? 

29. What more can you add to help us better 

understand the project in your 

state? 

For state project director only: 

We would like your help in identifying people 

at the local education agency (local education 

agency) level to interview. 

30. Have you worked more closely with districts 

or schools to implement the 

project? Give them the option of saying “both” 
as well. 

If response is districts or both: 

• Please consider three districts whose 
participating schools you believe 
demonstrate the strongest and most 
widespread implementation of the 

project. Which districts 

would you say have most fully imple- 
mented the project, in rank order? 

• In each of these districts, who, in 
rank order, would you say are the 
people most involved in and knowl- 
edgeable about the project implemen- 
tation? Probe for multiple names. Can 
you give us contact information? 



• For each of these three districts, 
please name one or two participating 
schools whose teachers you believe 
demonstrate the strongest and most 
widespread implementation. Or, do 
you think it would be better to ask 
district project leadership to recom- 
mend a school? 

If they provide school names: 

• Can you give us the names of people 
at those schools most involved in 
and knowledgeable about the project 
implementation? Or, do you think it 
would be better to ask district project 
leadership to recommend school- 
level leaders? 

If response is schools: 

• Please consider three schools whose 
teachers you believe demonstrate 
the strongest and most widespread 
implementation of your project. 
Which schools would you say have 
most fully implemented the project, 
in rank order? 

• In each of these schools, who, in rank 
order, would you say are the people 
most involved in and knowledgeable 
about the project implementation? 
Probe for multiple names. Can you 
give us contact information? 

• For each of the schools you men- 
tioned, can you give us the names of 
people in the school’s corresponding 
district who would be most knowl- 
edgeable about project implementa- 
tion in the district? Or, do you think 
it would be better to ask school 
project leadership to recommend a 
district-level leader? 
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VII. Verification and completion of data collection 
matrix (research question 1) 

Please help us complete or confirm information on 
your state project listed in this data matrix. 

Probe/review each category: 

a. Overarching project goals 

b. Definitions of formative assessment 

c. Envisioned classroom assessment 
practices 

d. Project structure (including staff roles, 
partner roles, target content areas, stu- 
dent grade levels, number of schools and 
districts, project timeline) 

e. Funding sources 

f Services and supports (including pro- 
fessional development and technical 
assistance) 

g. Monitoring and/or evaluation procedures. 
Thank you very much for your time! 



District initiative leaders 

Introduction 

Thank you for agreeing to speak with us today 
about your district’s efforts to promote formative 
assessment. 

We are [introduce selves], members of the Re- 
gional Educational Laboratory Northeast and 
Islands, funded by lES. 

We understand that your district has participated 

in [state’s] Project, and we have 

spoken with people at the state education agency 
(SEA) about how they have worked to promote 
formative assessment practices within the state. 
Our goals for this interview are to learn: 



a. How your district has worked with the 
state and other groups to implement the 
project and to promote forma- 
tive assessment practices, and 

a. How effective you think state and local efforts 
have been to implement the project. 

Do you have any questions before we begin? 

I. Individual background and district background 
in the project: 

First, we would like to learn about your involve- 
ment in project. 

1. What is your role in the district? 

2. What has your role been in [supporting formative 

assessment efforts/the project]? 

3. Now, we would like to learn a little about the 

district’s motivation for getting involved in 
the project. How did your dis- 

trict, and specific schools within your district, 
come to participate in the project? 

Probes: 

• How many schools within the district are 
participating in the project, and how were 
they selected? 

• What content areas and grades levels does 

the project target? 

• Is teacher participation in the 

project voluntary or required? 

• If voluntary, about what percentage 

of eligible teachers are participating 
in each school site? 

• How has the project been funded? 

• If not already answered: What benefits did 
your district hope to achieve by partici- 
pating in the project? 
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4. What has been the working definition of “for- 
mative assessment” within this project? 

Follow up: 

• Using this definition, what does success- 
ful formative assessment look like? In 
other words, what do you expect to see 
from teachers? Students? Parents? 

• Outside the scope of this project, does 
your district have additional expectations 
for what successful formative assessment 
looks like? 

If yes, follow up: 

• Please describe these expectations. 

• What has your district done to help 
schools and teachers achieve those 
additional expectations? 

IL Methods of project implementation 

Now we’d like to ask how the project 

has been implemented in your district. 

5. What are the primary supports that the dis- 
trict has provided to help schools and teachers 
carry out the work of the project? 

6. How enthusiastic would you say school and 
teacher buy-in have been for the goals and 
practices promoted by this project? 

Follow up if not addressed: What evidence 
illustrates this level of buy-in? 

7. How has the district monitored progress at 
the school or classroom level for the proj- 
ect? (Probe for measures besides student 
achievement) 

8. How have teachers been held accountable for 
fully implementing the formative assessment 
practices promoted by the project? Probe: In 



what ways do teacher practices related to the 
project factor into teacher evaluation? 

9. What have been the primary challenges for 
your district in implementing this project? 

10. How has the district addressed those 
challenges? 

Follow up, if not addressed: In your district, 
how have you worked to address potential 
school or teacher resistance to changes in 
school or classroom practice promoted by this 
project? 

Considering those inside and outside 
your district, who has provided the most 
support for implementing [formative 
assessment/the project] in your district? 

e.g., 

• The state 

• External partner(s) 

• District leadership 

• School leadership. 

If interviewee provides a list, choose the 
one they list or rank first, then ask: 

a. What types of support have they 
provided? 

b. In what ways have they been most 
helpful? 

Possible examples for b and c: increas- 
ing capacity to provide teacher sup- 
port; helping to negotiate competing 
demands. 

c. What could they have done dif- 
ferently, if anything, to help bet- 
ter implement this project in your 
district? 
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Probe series of questions a-c above for each 
person/agency/group also mentioned by 
interviewee. 

If not already discussed, ask series for each of 
the following: 

• State/SEA Probe for other SEA supports 
that might have built district capacity to 
participate fully in this project 

• External partner(s) 

• District leadership [if not covered in 
earlier discussion] 

• School leadership [if not mentioned in 
earlier discussion] 

11. How well has the work for this project been 
aligned with other programs or policies active 
in your district? 

Probes: 

• Can you give examples of other district 
programs or policies that support or com- 
pete with the work of this project? 

• Can you give examples of state guidance 
or requirements that support or compete 
with this project? 

Ill Perceived outcomes of project participation 

Now we’d like to ask a few questions about how 
participation in the project has affected your 
district. 

12. In what ways have teacher practices changed 
within participating schools in your district? 

Probes: 

• How has participation in the project 
changed: 

• Classroom instruction? 



• Assessment practices? 

• Content and structure of teacher 
meetings? 

• Teacher collaboration? 

• What evidence demonstrates these 
changes have occurred? 

• How have these changes affected 
students? 

13. You have just described specific changes in 
teacher and school practice. Do these exam- 
ples apply to most, some, or few participating 
teachers? 

14. So far, what have been the primary benefits to 
your district by [participating in this project/ 
promoting formative assessment practices]? 

Follow up if not addressed: 

• What have been the primary benefits to 
schools in your district? 

If benefits are covered in earlier response, 
ask which are primary. 

IV. Future directions 

15. What form do you think [efforts to support 
formative assessment/the project] will take in 
the next year in your district? The next 3-5 
years? 

Probe if not addressed: 

• How do you anticipate the project or 
related work will be funded? 

16. Knowing what you know now, what would you 
try to do differently to support formative as- 
sessment practices at schools in your district? 

17. The motivation for this study was a question 
posed by state education agency officials about 
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the best role the state can play to support School initiative leaders 



the classroom use of formative assessment, 
where teachers engage in a process of frequent 
assessment to adjust ongoing instruction to 
meet individual student learning needs. 

• How do you think the state can best 
support classroom-embedded formative 
assessment practices? 

• How about the district? What is its best 
role in supporting classroom-embedded 
formative assessment practices? 

V. School recommendations 

18. If school not yet selected: Please consider three 
schools you believe have made good progress 
toward full implementation of the project. 
Which three would you say have most fully 
implemented the project, in rank order? 

19. If school not yet selected: In each of these 
schools, who, in rank order, would you 
say are the people most involved in and 
knowledgeable about implementation of the 
project? Probe for multiple names and contact 
information. 

20. Can you tell us a little bit more about the 
school that we will be visiting? 

• Their history with this project. 

• Continuity of leadership and teaching 
staff? (Lots of turnover?) 

• Why this school? 

• How does their implementation of the 
project compare to that of other partici- 
pating schools in your district? 

Thank you very much for your time! 



Introduction 

Thank you for agreeing to speak with us today 
about your school’s efforts to promote formative 
assessment. 

We are [introduce selves], members of the Re- 
gional Educational Laboratory Northeast and 
Islands, funded by lES. 

We understand that your school has participated 

in [state’s] project, and we have 

spoken with people at the state department of 
education about how they have worked to promote 
formative assessment practices within the state. 

Our goals for this interview are to learn 

a. How your school has implemented the 

project and promoted formative 

assessment practices, and 

b. How participation in the project has changed 
teacher practice at your school. 

Do you have any questions before we begin? 

I. Background 

First, we have a couple of background questions 
about you. 

1. What is your role in the school? 

2. How did you get involved with [the project/ 
formative assessment efforts] at your school? 

II. Results of participation in project 

Now we have questions about how participation in 
the project has affected your school. 

3. In your school, what does formative as- 
sessment look like for participating teach- 
ers? What practices related to formative 
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assessment could an observer expect to see on 
a regular school day? In a regular week? 

Probes: 

• What kinds of assessments are the focus 
of this project? 

• How are these assessments analyzed? By 
whom? 

• How are results of analysis used? 

4. How, if at all, has this project [or efforts to 
promote formative assessment] changed 

a. Assessment practices at your school? 

b. Instructional practices at your school? 

c. How regularly scheduled meeting times 
are used in your school? 

d. Informal conversation or informal col- 
laboration about assessment and learning 
among teachers? Among teachers and 
students? 

e. Students’ experiences in the classroom? 
f The school improvement plan? 

5. We are curious about the degree of partici- 
pation from eligible teachers in the forma- 
tive assessment practices you described 
earlier. 

• For eligible teachers, is participation in 

the project voluntary or re- 

quired? If voluntary, about what percent- 
age of eligible teachers are participating at 
your school? 

• How enthusiastic would you say teacher 
and administrator buy-in has been at 
your school for the goals and practices 
promoted by this project? 



• You have already described specific 
changes in teacher and school practice. 

Do these examples apply to most, some, 
or few participating teachers in your 
school? 

6. What aspects of carrying out the project 
would you identify as most successful? Why? 

7. What aspects of carrying out the project 
would you identify as least successful? Why? 

III. School structures, supports, resources devoted to 
this project 

We now have some questions about how your 
school has structured and supported the work of 
this project. 

8. How have teachers been supported to carry 
out the work of the project? We are interested 
both in school-embedded and outside-school- 
day supports. 

Probe for structures and supports, such as 

• Meetings/formal collaboration; 

• Online resources; 

• Common planning time; 

• Training/coaching, 

• Collaboration with other schools. 

9. What professional development has been 
provided to teachers and school leaders to 
support this work? Please describe its content. 

10. What main challenges has your school en- 
countered as you have worked to change class- 
room assessment and instructional practices 
through the project? 

• Please give examples of how the school 
has addressed those challenges. 
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1 1 . What methods has your school employed to 
encourage participation in the project by all 
the teachers in the school? 

Follow up: 

• How has your school encouraged teachers 
to integrate practices from this project in 
daily work? 

• How has your school worked to overcome 
potential teacher resistance to changes 
in classroom practice promoted by this 
project? 

• How has your school worked to support 
teachers who have struggled or had dif- 
ficulty with changes in classroom practice 
promoted by this project? 

12. We are interested in how “regular ways of 
doing business” at this school have supported 
or complicated the work of this project. For 
example, how have the following factors af- 
fected the work of this project: 

• Structures, like school schedules and 
common planning time 

• Norms and staff practices, like how 
department/team/grade level/faculty 
meeting time is used 

• Leadership by formal administrators or 
informal teacher leaders 

• Culture of the school (nature of conver- 
sations and relationships among staff, 
between students and teachers). 

IV. Contributions from outside the school for this 
project 

Come prepared to probe with list of supports dis- 
cussed in previous interviews. 



13. Have you received direct support from the 
district for this project? 

If yes: 

• In what ways has the district been most 
helpful? 

• What could the district have done differ- 
ently, to better support your school’s work 
in this project? 

14. Have any external partners supported your 
school’s work in this project? 

If yes: 

• In what ways have these partners been 
most helpful? 

• What could the external partners have 
done differently, to better support your 
school’s work in this project? 

15. Have you received direct support from the 
state for this project? 

If yes: 

• In what ways has the state been most 
helpful? 

• What could the state have done differ- 
ently, to better support your school’s work 
in this project? 

16. What external supports have been most help- 
ful for carrying out the work of the project, 
and why? 

17. Has your school modified recommended 
approaches for this project to better fit the 
context of your school? If yes, how? 
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18. How much do other district programs or poli- 
cies complement or compete with the work of 
this project? Please describe. 

Follow up: 

• How much do other state guidance or re- 
quirements complement or compete with 
the work of this project? Please describe. 

V. Future directions 

19. What form do you think [efforts to support 
formative assessment/the project] will take 
in the next year in your school? The next 3-5 
years? 

20. In what ways do you and your school leader- 
ship plan to continue to support this project at 
your school? How about formative assessment 
practices? 



21. The motivation for this study was a request 
by state education agency officials about the 
role of the state in supporting the classroom 
use of assessment feedback, in which teach- 
ers adjust ongoing instruction to improve 
student achievement of targeted skills and 
understandings. 

How would you answer their question: how 
do you think the state can best support 
classroom-embedded formative assessment 
practices? 

Ask if time allows: How do you think the 
district can best support classroom-embedded 
formative assessment practices? 

22. What more can you add to help us better under- 
stand your school’s work related to the project? 

Thank you very much for your time! 
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APPENDIX F 

INTERVIEW DATA CODES 

Interviews were digitally recorded, transcribed, analysis software package ATLAS.ti. Table FI lists 

and electronically coded using the qualitative data the interview data codes by major code categories. 

TABLE FI 

Interviewdata codes for analysis of formative assessment initiatives in New York and Vermont, 2010 



H Code 


Definition 1 


Background 

Definition formative assessment 


Includes project history, scope, goals, who Is involved, and how got involved 

The major conceptual formative assessment code. Definition of formative assessment 
used within initiative; includes history/source, descriptions, and described outcomes of 
different understandings (including attempts to reach shared understanding). 


Description formative assessment 


Description of "what formative assessment looks like" in an ordinary day, week. Also 
includes roles for teachers/students/parents (even if vague). 


Evidence of success 


Type of evidence initiative uses to measure success. 


External, history 


Flow state education agency chose external partners/how external partners joined the 
project; also, which external partners supported the project. ("External" includes private 
vendors, universities, federal comprehensive center). 


Goals, project 


Stated goals for project. 


State education agency 
department-primary, history 


Flow primary state education agency department became involved with the formative 
assessment initiative, including relevant background. 


State education agency 
department-other, history 


Flow other state education agency departments became involved with the formative 
assessment initiative, and what their role was in the project, including relevant 
background. 


Target content 


Which content areas formative assessment project targeted. 


Target district 


Flow districts came to participate in project. 


Target grades 


Which grade levels formative assessment project targeted. 


Target school 
Context 


Flow schools came to participate in project. 

Prior or current conditions that describe the setting of the initiative 


Implementation 




Challenges/constraints 


Challenges faced to implement project, past or current/ongoing. [Flow challenges are 
dealt with is a separate code, under Method.] Also includes constraints (lack of capacity 
described by participants, such as lean staff, past reductions). 

[Does not include unengaged teachers, which is a separate code under Engagement.] 


Competition, district 


Other district initiatives, policies, or requirements that compete with or hinder project. 


Competition, state 


Other state initiatives, policies, or requirements that compete with or hinder project. 


Complement, district 


Other district initiatives, policies, or requirements that complement or support project. 


Complement, state 


Other state initiatives, policies, or requirements that complement or support project. 


School 


School characteristics, such as: 

• Student demographics, 

• Staff/leadership characteristics 

• Staff type and organization (for example, in-house coaches; vice principals) 

• School norms and common staff/student practices 

• Nature of staff relations 

• School climate 

• School structures (for example, schedules, common planning). 



Code Definition 



(CONTINUED) 
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TABLE FI (CONTINUED) 

Interviewdata codes for analysis of formative assessment initiatives in New York and Vermont, 2010 



Code Definition 



1 Code 


Definition | 


Context (continued) 




District 


District characteristics, such as: 

• Student demographics 

• District staff/leadership characteristics 

• Staff type and organization (forexample, four staff members) 

[Characteristics are of the district under study. Characteristics of districts in general are 
under separate code Role, District] 



General 



State 


Includes state-level organization (offices and departments), procedures, interactions with 
legislature and regents, etc. [Constraints/challenges are under separate code Context] 


Turnover 


Staff turnover at any level (state, district, school) that participant identifies as affecting 
project participation or implementation. 


Engagement 


Special code. Separate category because can be used to Indicate both the context of 
the Initiative (for example, people's receptivity to change) and an outcome (for example, 
how people react to the initiative). 

• Used to highlight descriptions of people (at state, district, school, or student levels) 
showing engagement or enthusiasm for the initiative and its practices. Includes 
demonstration of buy-in. 

• Also used to highlight descriptions of lack of engagement or enthusiasm for the 
initiative. 


Future 


• Can include discussion of reasons why described levels of engagement have arisen. 

[Does not include leadership commitment (engagement as method of implementation), 
which is under Method, promote targeted practices.] 

Future directions for work related to this project. Use for current practices related to the 
initiative that people plan to or will carry forward as they look ahead. 


Funding 


Future funding for coordinated formative assessment work. 


Plans, goals, methods, hopes 


Future plans (Including goals, methods, participants, and coordination) for formative 
assessment-related work in next year and next 3-5 years. Includes hopes for future work. 
Includes new incorporation. 


Method 


Methods of implementation of formative assessment initiative. Intentional strategies and 
actions used to carry out the goals and work of the Initiative. 


Promote targeted practices 


Strategies or actions (formal or informal) that individuals or groups have used to 
promote project's target practices. Includes policies making participation mandatory or 
voluntary, types of incentives offered, and how expectations are set and shown. 

• Leadership strategies (actions/messages of persistence, commitment to initiative that 
affect implementation). 

• Laundry list of items, broad strategy. 

[Where relevant, ok to double-code with Context, New, structures, or Professional 
development Promote targeted practices is the overall code; New and Professional 
development codes are subcategories.] 


Challenges, response 


Flow dealt/dealing with challenges to implement project [description of challenges Is 
separate code, under Context] 


Evaluation and data collection 


Methods of evaluating the progress and success of the initiative or of recording 
or documenting project activities for evaluation or other purposes (for example, 
professional development). Can be formal or informal. 



(CONTINUED) 
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TABLE FI (CONTINUED) 

Interviewdata codes for analysis of formative assessment initiatives in New York and Vermont, 2010 



Code Definition 



H Code 


Definition H 


Funding 


State's sources and methods for attaining initiative funding. 


Local modifications 


Flow recommended approaches were modified to fit local context. 


New, structures 


New events/structures created to carry out this project (for example, new formal 
meetings, schedules, roles). New structures have become systemic. Institutionalized, 
or regular parts of how the organization functions. Includes content and audiences 
targeted. 


Professional development, form 


Description of professional development provided to teachers and school leaders for the 
initiative. Includes description of structure/format (for example, workshops, professional 
learning communities, coaching) and audiences served. 


Professional development, content 


Description of content or the substantive material covered In the professional 
development delivered for the Initiative. Includes discussing best practices, sharing 
lesson plans, examining student work, analyzing data, and so on. 


Technical assistance 


Description of technical assistance provided for the project, specifically advice or 
consulting. Includes discussion of structure, content, and audiences targeted. 


Outcome 


Student, teacher, and school outcomes possibly due to project (includes district and 
state). 


School/district improvement plan 


Flow school/dlstrict Improvement plan has changed possibly due to project. 


Student achievement 


Flow student achievement has changed possibly due to project. 


Teaching and learning practices 


Flow specific teacher classroom practices (including Instruction, assessment) or student 
practices (for example, learning habits, teacher/peer interactions) changed possibly due 
to project. 

[For changed motivation/engagement In teaching or learning, use Engagement code.]. 


Increased knowledge, professional 
development 


Increased general knowledge of formative assessment due to project; increased ability 
to conduct or train people to conduct formative assessment. Applies to any level 
(Including state). 


Staff interactions 


Changes in how staff (teachers or administrators) talk and work with each other, formally 
and Informally, possibly due to project. 

[For new collaborative structures, such as professional learning community meetings, 
created to do the work of the project, use Method, new, structures.] 


Other or vague 


Use for outcomes that might have been due to project, not including student and 
teacher outcomes above. Includes vague statements. 

[For new structures created for project, use Method-new, structures; for what should be 
an outcome, use Role codes.] 
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NOTES 

1 . Data on principal participation in these ses- 
sions are not available. 

2. Data on state leaders’ participation in these 
sessions are not available. 

3. Although the pilot was officially a one- 
year initiative, participating schools and 
teachers were required to make a two-year 
commitment. 

4. Critical friends groups are professional 
learning communities of 8-12 educators 
who meet at least once a month with a focus 
on improving practice and student learning 
(www.nsrffiarmony.org/faq.html). Vermont 



schools under restructuring are given the 
option of creating critical friends groups if 
they choose not to participate in the VT-FAP 
initiative. 

5. An example of a coding difficulty involved the 
concept of teacher engagement in formative 
assessment, which was described multiple 
ways in the interview transcripts. Interview- 
ees variously described teacher engagement as 
a goal, implementation method, or outcome 
(current goal of the initiative, method of at- 
tracting other teachers, outcome of participa- 
tion, and future goal). Coders did not always 
agree on which of these categories to assign 
references to teacher engagement. To resolve 
this issue, coders made “engagement” a sepa- 
rate coding category. 
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